Black box model watermark generation and verification method and device and medium

Abstract

The invention relates to a black box model watermark generation and verification method and device and a medium, and the method comprises the steps: building a generator, inputting a preset label and a potential vector composed of random noise conforming to Gaussian normal distribution into the generator at the same time, and obtaining an image as a trigger; based on the trigger and the original training sample, performing fine tuning on a pre-trained target DNN model, updating parameters of the DNN model, and obtaining a DNN model containing watermarks; and using the generator, the potential vector and a preset label to generate a trigger sample, inputting the trigger sample into the to-be-verified model, obtaining the classification accuracy of the trigger sample, and realizing model verification. Compared with the prior art, the method has the advantages of being accurate in verification, small in performance influence, capable of avoiding watermark repeated verification and the like.

Description

Technical Field

[0001] The present invention relates to the field of encryption technology, and in particular to a black box model watermark generation and verification method, device and medium. Background Art

[0002] With the development of computer hardware conditions and big data, artificial intelligence technologies represented by deep neural networks (DNNs) have achieved remarkable achievements in many application fields such as image recognition, natural language processing and autonomous driving. Building these high-performance deep neural network models not only requires the wisdom of designers, but also consumes a large amount of training data and computing resources. Therefore, as an expensive digital asset, it is crucial to protect the intellectual property rights of deep neural network models from any form of infringement.

[0003] DNN watermarking technology has been widely studied for protecting the intellectual property rights of models. There are two main categories: white-box watermarking and black-box watermarking. During the verification phase, white-box watermarking requires obtaining the internal details of the victim DNN model, while black-box watermarking can directly access the DNN application programming interface (API). DNN is usually an online-based service that can only be accessed through an API, so black-box watermarking is more practical in daily life.

[0004] Recent research has focused on improving the performance of watermarking techniques, but relatively little work has been done to deal with verification-related issues. Most importantly, almost all state-of-the-art work is performed under the assumption that the verification operation is simple and is performed by the owner or a reliable verification authority. However, through the verification process, the watermarks are known to both the audited attacker and the verification authority, and the audited attacker can also submit the acquired trigger set to verify the copyright of the suspicious model, thus failing to guarantee the sole ownership verification of the model owner. Summary of the invention

[0005] The purpose of the present invention is to provide a black box model watermark generation and verification method, device and medium in order to overcome the defects of the above-mentioned prior art.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] A black box model watermark generation and verification method comprises the following steps:

[0008] Generation phase: A generator is built, and a potential vector consisting of a preset label and random noise that conforms to a Gaussian normal distribution is simultaneously input into the generator to obtain an image as a trigger;

[0009] Embedding stage: Based on the trigger and the original training sample, the pre-trained target DNN model is fine-tuned, the parameters of the DNN model are updated, and the DNN model with watermark is obtained;

[0010] Verification phase: Use the generator, latent vector and preset label to generate trigger samples, input the trigger samples into the model to be verified, and obtain the classification accuracy of the trigger samples. When the classification accuracy is greater than the threshold, the verification is successful.

[0011] Furthermore, the calculation expression of the objective function of the DNN model is:

[0012]

[0013] Where c' g is the predicted label of the trigger, c' is the label of the original sample, To generate sample c' g With c g The cross entropy loss between original is the cross entropy loss of the original sample, which indicates the classification performance of the target model C for the original training sample.

[0014] Furthermore, the calculation expression of classification accuracy is:

[0015]

[0016] In the formula, C s is the model to be tested, To trigger the sample, c' v is the preset label, 1 condition Represents the conditional function, which returns 1 or 0 when the condition is true or false, respectively, and p represents the pre-set logits distribution.

[0017] Furthermore, the generator consists of multiple transposed convolutional layers, a ReLU layer, multiple batch normalization layers, a ReLU layer, and a Tahn layer connected in sequence.

[0018] Furthermore, a discriminator is included, and the discriminator is used to determine whether the sample in the input model is an original training sample or a trigger.

[0019] Furthermore, the discriminator consists of multiple convolutional layers, multiple groups of first combination layers consisting of batch normalization layers, leaky relu layers and dropout layers connected in sequence, a second combination layer consisting of batch normalization layers and leaky relu layers connected in sequence, a sigmoid function layer and a softmax function layer.

[0020] Furthermore, the calculation expressions of the objective function of the generator and the objective function of the discriminator are:

[0021] L D =logB real +log(1-B fake)+logC real +logC fake

[0022]

[0023] In the formula, B real Represents the binary output when the original training sample is input into the discriminator, B fake represents the binary output when the trigger is input into the discriminator, C real represents the category possibility output result obtained when the original training sample is input into the discriminator, C fake represents the category possibility output result obtained when the trigger is input into the discriminator, L CG is the cross entropy loss between the predicted label of the trigger and the predefined label, L is the image type of the classification task, and P i is the probability distribution output by the classification model.

[0024] Furthermore, it also includes an invalid data set, where the trigger sample enters the invalid data set after testing, and when the trigger sample that has entered the invalid data set in the verification stage is input into the model to be verified again, verification is refused.

[0025] In a second aspect of the present invention, a black box model watermark generation and verification device includes a memory, a processor, and a program stored in the memory, and when the processor executes the program, it implements any one of the above black box model watermark generation and verification methods.

[0026] A third aspect of the present invention is a storage medium having a program stored thereon, which implements any of the above black box model watermark generation and verification methods when the program is executed.

[0027] Compared with the prior art, the present invention has the following beneficial effects:

[0028] 1) The present invention uses latent vectors and preset labels to generate triggers through a generative adversarial network (GAN), and embeds the backdoor watermark represented by the special trigger into the target DNN model through a fine-tuning method. According to the pre-set verification rules, each time a suspected model is found, the owner can effectively generate a new trigger sample, and verify the watermark DNN model through the classification results of the trigger sample by the suspected model, thereby protecting the watermark DNN model and avoiding a significant impact on the model performance.

[0029] 2) The present invention verifies the validity of the trigger sample through an invalid data set, and the used trigger sample cannot be used for verification again, thereby avoiding the misappropriation of the trigger sample for copyright verification. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a schematic diagram of the method flow of the present invention.

[0031] Figure 2 This is a test result diagram of the present invention. DETAILED DESCRIPTION

[0032] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0033] Example 1

[0034] The present invention is a black box model watermark generation and verification method. Figure 1 As shown, the following steps are included:

[0035] Generation phase: A generator is built, and a potential vector consisting of a preset label and random noise that conforms to a Gaussian normal distribution is simultaneously input into the generator to obtain an image as a trigger;

[0036] In order to build the generator, we first select the deep neural network models that need to be copyrighted and the corresponding original training sets. For example, the CIFAR-10 dataset is used as the original training set for the Resnet-18, Resnet-34, and Resnet-50 models, and the MNIST dataset is used as the original training set for the VGG13, VGG16, and VGG19 models.

[0037] ACGAN is used to generate triggers and trigger samples. A latent vector z composed of random noise from a Gaussian normal distribution and a preset label c are g At the same time, the image T is input into the generator G. w will be generated as a trigger. In addition, the trigger is classified by the pre-trained DNN model to obtain the predicted label c v .

[0038] We use ACGAN to generate the trigger set. The ACGAN used consists of two parts: the generator G and the discriminator D. The input of the generator G is the preset label and the latent vector. The preset label is embedded, and then the result of the operation and the latent vector are multiplied. The generator G consists of 5 transposed convolutional layers, a ReLU layer, three groups of batch normalization layers, a ReLU layer, and a Tahn layer. The final output is the generated image. On the other hand, the discriminator D consists of 4 convolutional layers, followed by 3 pairs of first combination layers, including batch normalization layers, leakyReLU layers, and dropout layers, and a pair of second combination layers, including batch normalization layers and leakyReLU layers, and finally a sigmoid(·) function layer for distinguishing the authenticity of the input generated image, and a softmax(·) function layer for obtaining the predicted label of the input generated image.

[0039] A latent vector z consisting of random noise from a Gaussian normal distribution and a preset label c are combined g At the same time, the image T is input into the generator G. w will be generated as a trigger. In addition, the trigger is classified by the pre-trained DNN model to obtain the predicted label c v In order to obtain a specific output, the confidence score of the classification result needs to satisfy a predefined distribution. Therefore, the probability distribution P of the penultimate layer of the target classification model C is adopted. i To represent the specific output distribution of the input trigger. Through the discriminator D, it is distinguished whether the input data belongs to the trigger or the original training sample. The calculation expressions of the objective function of the generator and the objective function of the discriminator are:

[0040] L D =logB real +log(1-B fake )+logC real +logC fake

[0041]

[0042] In the formula, B real Represents the binary output when the original training sample is input into the discriminator, B fake represents the binary output when the trigger is input into the discriminator, C real represents the category possibility output result obtained when the original training sample is input into the discriminator, C fake represents the category possibility output result obtained when the trigger is input into the discriminator, is the cross entropy loss between the predicted label of the trigger and the predefined label, L is the image type of the classification task, and P i is the probability distribution output by the classification model.

[0043] Embedding stage: Based on the trigger and the original training sample, the pre-trained target DNN model is fine-tuned, the parameters of the DNN model are updated, and the DNN model with watermark is obtained;

[0044] Based on the latent vector z and the pre-set label c g The generator G generates trigger samples and the original training samples with corresponding labels c, fine-tunes the pre-trained target DNN model C to minimize the impact on the original task performance, and then updates the parameters of C. The predicted label of the trigger is represented as c′ g , the label of the original sample is c'. The objective function corresponding to C is as follows.

[0045]

[0046] Where c' g is the predicted label of the trigger, c' is the label of the original sample, To generate sample c' g With c g The cross entropy loss between original is the cross entropy loss of the original sample, which indicates the classification performance of the target model C for the original training sample.

[0047] Verification phase: Use the generator, latent vector and preset label to generate trigger samples, input the trigger samples into the model to be verified, and obtain the classification accuracy of the trigger samples. When the classification accuracy is greater than the threshold, the verification is successful.

[0048] When a suspicious model C is found s When the ownership verification process is performed by the verification agency V, if the model is found to be illegally used, the model owner R will verify the ownership of the model based on the latent variable z and the preset label c' v Generate trigger samples through the trained trigger sample generator G And submit a verification request to the verification agency V. The verification agency V verifies the suspected model C through the API based on the submitted trigger sample. s Verify. Classification accuracy Acc wm The calculation method expression is:

[0049]

[0050] In the formula, 1 condition represents a conditional function that returns 1 or 0 when the condition is true or false, respectively. p represents a pre-set logits distribution. wm When it is greater than the threshold ∈, the verification agency can declare that this suspicious model C s Stolen, the model belonged to R.

[0051] The copyright verification process contains two key steps: 1) the predicted label is consistent with the pre-set trigger sample label; 2) a 1-bit watermark message is obtained from the predicted probability distribution. Only when both steps are met, the ownership verification is successful. After the owner verifies the copyright using the trigger set, these images are placed in the invalid dataset. In the next verification stage, the verification agency will check the validity of the submitted trigger samples by comparing them with this dataset. If the trigger sample is valid, the submitted trigger sample will be input into the suspicious model and its ownership will be verified. If the trigger sample is invalid, these trigger samples are denied access and the copyright verification fails. The verified trigger samples cannot be delivered to and used in the next verification stage.

[0052] In order to verify the watermark protocol effect of the above solution of the present invention, relevant test experiments were also carried out. The experimental results are shown in Table 1 and Figure 2 shown.

[0053] Table 1 Classification accuracy of the original model and the watermarked model on the test set and trigger set respectively

[0054]

[0055] As shown in Table 1, the experiments show that the test classification accuracy of the original training set has no significant decrease, and even slightly improves, which shows that the fidelity of the proposed model watermarking protocol can be guaranteed. Therefore, the resulting watermarked DNN model can obtain high verification accuracy of the generated trigger samples without affecting the classification accuracy of the original task.

[0056] like Figure 2 As shown in Figure 2, the test results of different watermarked models against pruning attacks on the original classification task and watermark verification task, where the watermarked DNN model is executed Pruning method, where the pruning rate refers to the percentage of pruned parameters, the pruning rate ranges from 0 to 80%, and the step size is set to 10%. The test results show that even if the pruning rate is at a fairly high level, the verification performance of the attacked watermark model can still be considered acceptable. Therefore, the proposed DNN watermarking protocol can effectively resist the pruning attack of the model.

[0057] The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0058] In the context of the present invention, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0059] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A black box model watermark generation and verification method, characterized in that: The following steps are involved: Generation phase: A generator is built, and a potential vector consisting of a preset label and random noise that conforms to a Gaussian normal distribution is simultaneously input into the generator to obtain an image as a trigger; Embedding stage: Based on the trigger and the original training sample, the pre-trained target DNN model is fine-tuned, the parameters of the DNN model are updated, and the DNN model with watermark is obtained; Verification phase: Use the generator, latent vector and preset label to generate trigger samples, input the trigger samples into the model to be verified, and obtain the classification accuracy of the trigger samples. When the classification accuracy is greater than the threshold, the verification is successful.

2. A black box model watermark generation and verification method according to claim 1, characterized in that: The calculation expression of the objective function of the DNN model is: Where c' g is the predicted label of the trigger, c' is the label of the original sample, To generate sample c' g With c g The cross entropy loss between original is the cross entropy loss of the original sample, which indicates the classification performance of the target model C for the original training sample.

3. A black box model watermark generation and verification method according to claim 1, characterized in that: The calculation expression of the classification accuracy is: In the formula, C s is the model to be tested, To trigger the sample, c' v is the preset label, 1 condition Represents the conditional function, which returns 1 or 0 when the condition is true or false, respectively, and p represents the pre-set logits distribution.

4. A black box model watermark generation and verification method according to claim 1, characterized in that: The generator is composed of multiple transposed convolutional layers, a ReLU layer, multiple batch normalization layers, a ReLU layer and a Tahn layer connected in sequence.

5. A black box model watermark generation and verification method according to claim 1, characterized in that: It also includes a discriminator, which is used to determine whether a sample in the input model is an original training sample or a trigger.

6. A black box model watermark generation and verification method according to claim 5, characterized in that: The discriminator includes multiple convolutional layers, multiple groups of first combination layers composed of batch normalization layers, leaky relu layers and dropout layers connected in sequence, a second combination layer composed of batch normalization layers and leaky relu layers connected in sequence, a sigmoid function layer and a softmax function layer.

7. A black box model watermark generation and verification method according to claim 5, characterized in that: The calculation expressions of the objective function of the generator and the objective function of the discriminator are respectively: L D =logB real +log(1-B fake )+logC real +logC fake In the formula, B real Represents the binary output when the original training sample is input into the discriminator, B fake represents the binary output when the trigger is input into the discriminator, C real represents the category probability output result obtained when the original training sample is input into the discriminator, C fake represents the category possibility output result obtained when the trigger is input into the discriminator, is the cross entropy loss between the predicted label of the trigger and the predefined label, L is the image type of the classification task, and P i is the probability distribution output by the classification model.

8. A black box model watermark generation and verification method according to claim 1, characterized in that: It also includes an invalid data set. The trigger sample enters the invalid data set after testing. When the trigger sample that has entered the invalid data set in the verification stage is input into the model to be verified again, verification is rejected.

9. A black box model watermark generation and verification device, comprising a memory, a processor, and a program stored in the memory, characterized in that: When the processor executes the program, a black box model watermark generation and verification method as described in any one of claims 1-8 is implemented.

10. A storage medium having a program stored thereon, characterized in that: When the program is executed, a black box model watermark generation and verification method as described in any one of claims 1-8 is implemented.

Images