Deep neural network copyright protection method based on interpretable intrinsic features

Abstract

The application discloses a deep neural network copyright protection method based on an interpretable intrinsic feature, and belongs to the fields of cyberspace security and artificial intelligence security. A certain number of pictures are selected to form a verification set for verifying ownership; a fingerprint set representing intrinsic features of an original model and a test model is generated by using deep Taylor decomposition; and the similarity of the fingerprint sets of the original model and the test model is compared by using structural similarity, so that it is judged whether the test model is a pirated model. The application utilizes the uniqueness and robustness of the intrinsic features of the model, can effectively verify the ownership of the model under various model modifications and attack operations, and has interpretability. The application not only guarantees the robustness of the model ownership verification, but also meets the demand of people for interpretability of the ownership verification.

Description

Technical Field

[0001] This invention can effectively verify the copyright owner of a deep neural network (DNN). It is a method for protecting the copyright of deep neural networks and falls under the fields of cyberspace security and artificial intelligence security. Background Technology

[0002] In recent years, deep learning technology has been widely applied in various fields, including some security-related areas. Training high-precision deep learning models requires massive amounts of training data, substantial hardware computing resources, and expert knowledge; the training process often takes several weeks, making it extremely costly. Furthermore, a high-performance deep learning model can bring significant economic benefits; therefore, in recent years, deep learning models have been considered intellectual property. To save the cost of training a high-performance deep learning model, some attackers may illegally pirate the model for economic gain and claim ownership. Attackers can use various malicious methods to achieve their piracy goals. This is a serious intellectual property infringement and causes significant economic losses to the model owner.

[0003] Traditional multimedia watermarking is a type of hidden data embedded in multimedia carrier data, widely used in the field of multimedia copyright protection. However, digital watermarking in the multimedia field cannot be directly applied to the copyright protection of deep neural networks. In recent years, many researchers have begun to study watermarking methods for copyright protection of deep neural network models. Currently, existing watermarking methods for deep neural network models mainly include parametric watermarking, backdoor watermarking, and fingerprint watermarking. Parametric watermarking embeds the watermark into the parameters of the deep neural network model using regularization methods during the training phase, and performs copyright verification by extracting the watermark. Backdoor watermarking uses specific images (backdoor watermark images) to train the model and embed the watermark during the training phase. The model owner can trigger the backdoor to output a specific target class through specific backdoor inputs, thereby performing copyright verification of the deep neural network. Fingerprint watermarking mainly uses the uniqueness of the model's decision boundary for copyright verification. It mainly finds data points on the decision boundary as the model's fingerprint. During the verification phase, these data points are used for query verification to achieve the purpose of verifying copyright.

[0004] However, existing deep learning model watermarks are vulnerable to various attacks. Attackers can remove watermarks through operations such as fine-tuning model parameters, pruning the model, and reverse engineering. Meanwhile, with the continuous development of deep learning technology, there is a growing pursuit of interpretability in intelligent decision-making. Existing copyright protection methods for deep learning models lack interpretability. This patent innovatively proposes a robust deep neural network copyright protection method with interpretability, demonstrating promising research prospects and application value. Summary of the Invention

[0005] The purpose of this invention is to propose a robust deep neural network copyright protection method based on interpretable inherent features. This invention does not embed watermarks into the model (i.e., it does not modify the model), but uses interpretable methods to extract the model's inherent features as its fingerprint. Furthermore, this invention is robust to attacks such as model modification.

[0006] The objective of this invention is achieved through the following technical solution: a copyright protection method based on interpretable inherent features of a deep neural network, comprising the following steps: constructing a dataset for verifying copyright; preprocessing the dataset and initializing the model; training a deep neural network model; generating a certain number of fingerprints representing the inherent features of the model, using the generated fingerprints as the fingerprint set of the model; comparing whether the similarity between two generated fingerprint sets is greater than a threshold T, and finally determining whether the test model is a pirated model.

[0007] To further explain, the method for constructing a dataset for verifying copyright is as follows:

[0008] A certain number of images are selected as the dataset for copyright verification. n images are randomly selected from this dataset as the verification set X. v ={x (1) x (2) , ..., x (n)} — A dataset used to generate fingerprints when verifying model copyright.

[0009] To further explain, the methods for preprocessing the dataset and initializing the model are as follows:

[0010] Normalize the images in the dataset (that is, transform the pixel values ​​of the images from 0 to 255 to between 0 and 1), select an appropriate optimizer, and set an appropriate learning rate and number of iterations.

[0011] To further explain, the operation method for training a deep neural network model is as follows:

[0012] In this invention, the model being validated is an image classification model, namely G: X→Y, where X represents the set of images input to the model, and Y represents the set of labels corresponding to the images. During training, the difference between the probability of the output class label and the probability of the true label is calculated, and this difference is used as the loss for backpropagation. The model parameters are iteratively updated until the class label output by the model matches the true label. After training, the prediction accuracy of the model remains almost unchanged.

[0013] To further explain, the method for generating fingerprints representing a certain number of inherent features of the representation model is as follows:

[0014] Load the validation dataset X v ={x (1) x (2) , ..., x (n) Input image x into the trained deep neural network model G. (i) It also uses deep Taylor decomposition to generate an interpretation for each image, namely a fingerprint representing the inherent features of the model. i All images from the validation set are input into the model, and the corresponding fingerprint set S = {s1, s2, ..., s3} is generated using the deep Taylor decomposition method. n} and S′={s′1,s′2,...,s′ n The process of generating fingerprints can be represented by the following formula:

[0015]

[0016] G(x) is a prediction function learned in a deep neural network model, and this function is differentiable. Indicates at the root point (Right now: An approximate Taylor expansion is performed at (). This means that the gradient of the corresponding pixel p in the image is used as the correlation score between pixel p and the output. The correlation scores of all pixels are combined into a correlation matrix according to their positions, and the output is visualized as a saliency map—a fingerprint. This can be expressed by the following formula:

[0017] s i =Image(G(x) (i) ))

[0018] Image(·) represents a visualization operation for the interpretation of model decisions, i.e., generating the corresponding fingerprint.

[0019] To further explain, the method for determining whether the test model is a pirated model by comparing the similarity between the two generated fingerprint sets to see if it exceeds a threshold T is as follows:

[0020] Structural similarity (SSIM) is used to compare the similarity between S′ and S. SSIM(S′, S) compares the fingerprints generated from the same input image in S′ and S. SSIM(x, y) is a metric that measures the structural similarity between two images; in this method, it is used to compare the structural differences between the fingerprint representing the original model and the fingerprint representing the test model. The structural similarity SSIM(x, y) can be expressed by the following formula:

[0021]

[0022] Where λ x and λ y δ represents the average values ​​of the x and y values ​​of the image, respectively. xy δ represents the covariance of images x and y. x and δ y Let x and y represent the standard deviations of images x and y. This invention uses SSIM(x, y) to evaluate the similarity between the fingerprints generated by the original model and the test model, representing their inherent features, to determine whether the test model is a copy of the original model. If the structural similarity SSIM(S′, S) of the fingerprint sets of the original model and the test model is greater than T, then the test model is a copy; if SSIM(S′, S) is less than T, then the test model is not a copy.

[0023] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0024] (1) No modifications or watermarking are required to the model. This method overcomes the limitations of previous methods that required watermarking and subsequent extraction for copyright verification during the verification phase. Instead, it generates the model's inherent features as fingerprints during the verification phase and performs corresponding comparisons to achieve copyright verification. This method 1) only requires training according to the conventional deep neural network model training method, without adding any extra operations or modifying the model; 2) only requires generating fingerprints representing inherent features using deep Taylor decomposition during the verification phase, and achieving copyright verification of the deep neural network model by comparing the structural similarity between the fingerprint set of the original model and the fingerprint set of the test model.

[0025] (2) This invention utilizes the uniqueness of model decision interpretations to verify the copyright of the model. We refer to the interpretation of model decisions as the model's fingerprint. Since it is related to the internal parameters of the model, its fingerprint will show small differences after the model is modified. Therefore, we use the high similarity of fingerprints of the same models (including unmodified models and modified models) to perform copyright verification. This process does not require additional computational resources; it only requires generating the interpretation of model decisions during the verification phase, with a time complexity of O(n).

[0026] (3) This invention is robust to model modifications or attacks such as fine-tuning, pruning, and model compression. This method proposes a fingerprint that represents the inherent characteristics of the model, and the fingerprint can still be successfully verified after the model has undergone attacks such as fine-tuning and pruning.

[0027] (4) After a pirate steals a model, regardless of the type of watermark (backdoor watermark, parametric watermark, etc.) embedded in the model to claim ownership of the deep neural network model, if the structural similarity of the fingerprint set is greater than T, we can claim that the model is a pirated model. Because the interpretable fingerprints generated from the same image in the original model and its corresponding modified model are highly similar, this invention is robust against model theft attacks.

[0028] (5) Many decision-makers need to prove that their decisions are correct and fair. Currently, there is no interpretable method for copyright protection of deep learning models. In the verification stage, this invention utilizes an interpretable artificial intelligence method—deep Taylor decomposition—to extract inherent features as the fingerprint of the model. It leverages the property of deep Taylor decomposition—deep Taylor decomposition generates a unique explanation for a decision of the model, which serves as an inherent feature of the model. The inherent features representing the model are visualized as fingerprints. The fingerprint generation process has undergone rigorous theoretical derivation, and experimental results verify that the fingerprint can uniquely represent the model. Therefore, the decision to determine whether the test model is a pirated model is fair and correct. This deep neural network copyright protection method is interpretable. This invention is the first interpretable deep neural network copyright protection method. Attached Figure Description

[0029] Figure 1 Technical solution flowchart

[0030] Figure 2 Fingerprint examples from different models

[0031] Figure 3 Fingerprint instances formed by the same model after different operations Detailed Implementation

[0032] The following will combine Figure 1 The technical solutions of the present invention will be further described in detail below. It is obvious that the presented embodiments are only a part of the embodiments of the present invention and do not include all embodiments. The embodiments presented below are for further illustrating the present invention. It should be understood that the scope of protection of the present invention includes but is not limited to these embodiments, and should include all embodiments obtained using the present invention without any inventive effort.

[0033] The specific implementation of the present invention can be divided into the following steps:

[0034] Step 1: This invention first randomly selects a portion of data from the dataset to construct a validation set X. v ={x (1) x (2) , ..., x (n)}

[0035] Step 2: Preprocess the dataset by normalizing the images. Initialize the model by selecting an appropriate loss function, optimization method, learning rate, and number of iterations to train the target model.

[0036] Step 3: Input the validation set into the original model and use the deep Taylor decomposition method to generate a fingerprint set S = {s1, s2, ..., s...} that represents the inherent features of the original model. n The validation set is input into the test model, and a fingerprint set S′={s′1,s′2,...,s′} is generated using the deep Taylor decomposition method to characterize the inherent features of the test model. n}

[0037] Step 4: Use Structural Similarity (SSIM) to compare the similarity between S and S′. SSIM(S′, S) compares the fingerprints in S and S′. Figure 2 The paper demonstrates fingerprints generated when the same image is input into ResNet-18 and VGG-16 models, with a fingerprint similarity of 0.731. This invention empirically sets a threshold T (0.850). Fingerprints generated by the same model are visually indistinguishable, with their intrinsic feature similarity approaching 1. However, fingerprints generated by different models after deep Taylor decomposition show visual differences in their intrinsic feature contours. Furthermore, comparing fingerprint pixel values ​​reveals that the structural similarity of the intrinsic features represented by different models is much lower than the threshold. If the average structural similarity of the fingerprint set is greater than T, it proves that the test model is a pirated model; if the average structural similarity is less than T, it proves that the test model is not a pirated model.

[0038] The effectiveness and efficiency of the method of the present invention are verified through the following experiments:

[0039] Example 1: The deep neural network model used in this experiment is the VGG-16 image classification model, and the dataset is ImageNet. This dataset contains 1000 classes, from which 100 classes were randomly selected for the experiment. One image was randomly selected from each class to form the validation set, i.e., 100 images for the validation experiment. The learning rate was set to 0.001, the training iterations were 50 rounds, and the Adam optimization algorithm was chosen. Table 1 shows the structural similarity comparison of the fingerprints generated by the VGG-16 classification model after no model modification, fine-tuning, pruning, and watermark overlay attacks.

[0040] Without any modifications, the structural similarity of the fingerprint sets generated by the model is 1.000, indicating that the model can be uniquely represented by fingerprints.

[0041] In practice, model pirates often employ watermark removal attacks, such as fine-tuning and pruning, to remove potential watermarks from models. These operations modify the model's weights, resulting in fingerprint structural similarity less than 1. However, because these operations aim to maintain the model's prediction accuracy, these modifications are minimal. Even after these modifications, the generated fingerprint still maintains a similarity to the original fingerprint above a threshold T.

[0042] In this experiment, the model was trained for 50 rounds using 40% of the test set data to fine-tune it. Pruning mainly involved setting a certain proportion of neurons with smaller weights to zero. After fine-tuning and pruning, the structural similarity of the model's fingerprint set remained close to 1, indicating that the test model was pirated, meaning the model's copyright was successfully verified. This demonstrates that this invention can effectively verify the copyright of deep neural network models.

[0043] In reality, thieves might use watermark overlay attacks to claim ownership of pirated models. This invention tests the impact of backdoor watermarking on model copyright verification. By embedding Gaussian noise as a backdoor watermark into the model, noisy inputs produce specific outputs. During the watermark embedding process, a small number of model weights are modified. After embedding the watermark into the model, the structural similarity of the fingerprint set is 0.991, approaching 1 and much greater than the threshold T, indicating successful verification of the pirated model.

[0044] To better demonstrate the experimental results of this invention, Figure 3This document presents example images of fingerprints generated when the validation set is input into a VGG-16 classification model. The fingerprints are obtained without modification, after fine-tuning, pruning, and a watermark overlay attack. As shown in the figures, after fine-tuning, pruning, and embedding a backdoor watermark, the fingerprint representing the inherent features of the original model is visually indistinguishable from the fingerprint representing the inherent features of the test model. The changes are negligible after calculation using the SSIM algorithm. Therefore, the copyright verification method for the deep neural network model of this invention is robust.

[0045] Table 1. Experimental Results of Example 1

[0046]

[0047] Example 2: The deep neural network model used in this experiment is the ResNet-18 image classification model, and the dataset is ImageNet. 100 classes were randomly selected from 1000 classes for the experiment, and one image was randomly selected from each class as the validation set, i.e., 100 images were used for the validation experiment. The learning rate was set to 0.001, the training iterations were 50 rounds, and the Adam optimization algorithm was chosen.

[0048] Table 2 shows the structural similarity comparison of fingerprints generated by the ResNet-18 image classification model after no model modification, fine-tuning, pruning, and watermark overlay attacks. When the model is not modified, the fingerprints generated by the ResNet-18 image classification model are consistent; that is, multiple generated fingerprints are identical.

[0049] After fine-tuning and pruning, the fingerprint set generated by the ResNet-18 model can verify whether the test model is a pirated model. Even when the model is subjected to a watermark overlay attack, resulting in changes to the model's parameters, the fingerprint set generated by the ResNet-18 model can still verify whether the test model is a pirated model. Therefore, the copyright protection method for the deep neural network model of this invention is robust.

[0050] Table 2. Experimental Results of Example 2

[0051]

[0052] The above two embodiments are detailed descriptions of the present invention, but it should not be considered that the present invention is limited to the above two embodiments. For those skilled in the art related to the present invention, any simple deductions and substitutions made based on the concept of the present invention should be considered to fall within the protection scope of the present invention.

Claims

1. A copyright protection method based on interpretable inherent features of a deep neural network, characterized in that: It includes the following steps: Step 1: The present invention first requires randomly selecting a portion of data from the dataset to construct a validation set. X v ={x (1) ,x (2) ,...,x (n) }; Step 2: Preprocess the dataset and initialize the model; Step 3: Input the validation set into the model and use the deep Taylor decomposition method to generate saliency maps, which serve as fingerprint sets S and S′ representing the inherent features of the model; Step 4: Use Structural Similarity (SSIM) to verify ownership.

2. The copyright protection method based on interpretable inherent features of a deep neural network according to claim 1, characterized in that: Preprocessing operations on the dataset include: Normalize the images in the dataset by converting the pixel values ​​of the images from 0 to 255 to between 0 and 1.

3. The copyright protection method based on interpretable inherent features of a deep neural network according to claim 1, characterized in that: The initialization operations for the model include: Choose an appropriate loss function, optimization method, learning rate, and number of iterations to train the target model.

4. The copyright protection method based on interpretable inherent features of a deep neural network according to claim 1, characterized in that: The fingerprint sets S and S′ that generate the representation of the inherent features of the model include: Step 3.1 The calculation formula for generating the inherent features of the model is as follows: G(x) is the prediction function learned in the deep neural network model, and this function is differentiable. Indicates at the root point Right now: Perform an approximate Taylor expansion at that point. This means that the gradient of the corresponding pixel p in the image is used as the correlation score between pixel p and the output, which is the resulting interpretation of that pixel. Step 3.2 Combine the correlation scores of all pixels with the output according to the pixel positions into a correlation matrix, and visualize the output as a saliency map, as follows: s i =Image(G(x (i) )) x (i) It is the verification set X v The i-th image in G(x) (i) ) represents model G for image x (i) The prediction results, Image(·) indicates that the interpretation of the model decision will be visualized, that is, the corresponding fingerprint will be generated; Step 3.3 Input the validation set into the original model and use the deep Taylor decomposition method to generate a fingerprint set S = {s1, s2, ..., s...} that represents the inherent features of the original model. n }; Step 3.4 Input the validation set into the suspected piracy model and use the deep Taylor decomposition method to generate a fingerprint set S′={s′1,s′2,...,s′ n } 5. The copyright protection method based on interpretable inherent features of a deep neural network according to claim 1, characterized in that: Ownership verification using structural similarity (SSIM) includes: SSIM(S′,S) compares the intrinsic features generated from the same input image in S′ and S. The SSIM calculation formula is as follows: Where λ x and λ y δ represents the average values ​​of the x and y values ​​of the image, respectively. xy δ represents the covariance of images x and y. x and δ y This represents the standard deviation of the x and y values ​​of the image. SSIM(x,y) is used to evaluate the similarity between the saliency maps representing the inherent features generated by the original model and the suspected pirated model, thereby determining whether the suspected pirated model is an illegally obtained model. T is the fingerprint similarity threshold used to determine whether a model is a pirated model. If the structural similarity SSIM(S′,S) of the saliency map sets of the original model and the suspected pirated model is greater than T, it means that the suspected pirated model is a pirated model of the original model. If SSIM(S′,S) is less than T, it means that the suspected pirated model is not a pirated model of the original model.

Images