A personalized model copyright protection method and system based on DWT

Abstract

The present invention discloses a personalized model copyright protection method and system based on DWT, which relates to the field of computer science and technology, including: using a generation network G to generate a personalized specific pattern for an original image. Then, the original image and the specific pattern are converted into low-frequency and high-frequency areas respectively through DWT. Then, the high-frequency information of the specific pattern is embedded into the wavelet domain of the clean image as a watermark trigger, and the combined image is converted from the wavelet domain to the spatial domain through the inverse discrete wavelet transform IDWT, and finally a watermark sample is generated. Starting from the characteristic attributes of the sample, the present invention uses a feature extractor to calculate the cosine similarity between the watermark sample and the original sample on the basis of ensuring the diversity of the watermark sample set, and selects samples with high contribution to the watermark model, so as to make the watermark verification process more efficient and accurate.

Description

Technical Field

[0001] The present invention relates to the field of computer science and technology, and in particular to a DWT-based personalized model copyright protection method and system. Background Art

[0002] As a means of protecting deep neural networks (DNNs), model watermarking aims to help developers prove the ownership of models and prevent unauthorized copying, tampering, or abuse by embedding identification information into the models. Model watermarking can be divided into white-box and black-box methods according to the watermark verification method. In the white-box method, the verifier has full access to the internal structure of the model, such as weights, feature maps, and parameters, so as to embed watermarks into these elements. In the black-box method, the verifier can only access the input and output information of the model. In practical applications, since it is usually difficult for the verifier to access the internal details of the target DNN, black-box DNN watermarking has stronger applicability and has been widely studied.

[0003] Most black-box DNN watermarking methods build a specific backdoor trigger to establish a secret association between the backdoor pattern and a specific label, so that the model behaves normally when processing regular data, but predicts an abnormal label when encountering trigger data, thereby verifying the ownership of the model. However, most existing black-box watermarking methods rely on visible and universal triggers, which allows model stealers to determine the existence of watermarks by detecting significant features or identifying fixed patterns. In addition, existing black-box watermarking methods usually embed watermarks into a random selection of samples in the training data, without fully considering how to optimize sample selection. Summary of the invention

[0004] In order to solve the deficiencies mentioned in the above background technology, the purpose of the present invention is to provide a personalized model copyright protection method and system based on DWT.

[0005] In a first aspect, the purpose of the present invention can be achieved by the following technical solution: a personalized model copyright protection method based on DWT, the method comprising the following steps:

[0006] Receive a clean sample set, input the original samples in the clean sample set into the pre-established generative network model, output a specific pattern sample, perform domain conversion on the original samples and the specific pattern sample through discrete wavelet transform DWT and inverse discrete wavelet transform IDWT, and finally obtain the initial watermark sample set;

[0007] The initial watermark samples and the original samples in the initial watermark sample set are input into the pre-trained feature extractor, and the cosine similarity between the initial watermark features and the original features is output, and the initial watermark samples are sorted based on the cosine similarity to obtain a rough watermark sample set, and the rough watermark sample set is screened to obtain a final watermark sample set;

[0008] The final watermark sample of the final watermark sample set is input into the pre-trained suspicious model, and the watermark extraction success rate is calculated. If the watermark extraction success rate is greater than the set threshold, the suspicious model is determined to be a pirated model, otherwise it is determined to be a non-pirated model.

[0009] In combination with the first aspect, in some implementations of the first aspect, the method further includes: a calculation process of the specific pattern sample:

[0010] Each original sample x in the clean sample set is input into the generative network model G to generate a personalized specific pattern related to the input:

[0011] .

[0012] In combination with the first aspect, in some implementations of the first aspect, the method further includes: generating the network model G satisfies the following conditions:

[0013]

[0014] Among them, x is the original sample, and are two different samples in the clean sample set, and the Manhattan distance L1 norm is introduced as the distance metric. and , and the corresponding output modes are and , Manhattan distance is defined as:

[0015]

[0016] Where n is the dimension of the generated network output, and Respectively indicate the output mode and The kth component of , the Manhattan distance We measured different samples and After generating two modes of network output and The sum of the absolute differences in all dimensions;

[0017] The difference based on Manhattan distance must satisfy the following formula:

[0018] such that ,in,

[0019] in, is a positive threshold indicating the minimum acceptable distance between output patterns, such that there exists a threshold that satisfies this formula.

[0020] In combination with the first aspect, in some implementations of the first aspect, the method further includes: the process of performing domain conversion on the original samples and the specific pattern samples by discrete wavelet transform DWT and inverse discrete wavelet transform IDWT is as follows:

[0021] Use discrete wavelet transform DWT to transform the original samples and specific modes Convert from the spatial domain to the wavelet domain:

[0022]

[0023]

[0024] in, They refer to the four frequency subbands into which the original sample x is decomposed by DWT, representing the low-frequency part, the horizontal high-frequency part, the vertical high-frequency part, and the diagonal high-frequency part respectively.

[0025] same, They refer to the four frequency sub-bands into which a specific mode p is decomposed by DWT, respectively representing the low-frequency part, the horizontal high-frequency part, the vertical high-frequency part and the diagonal high-frequency part.

[0026] The high-frequency region HH of the specific pattern p is injected into the high-frequency region of the original sample x as a watermark trigger;

[0027] Before injection, the high-frequency region HH of a specific mode p is fine-tuned: the HH region is rotated at different angles around the center point to generate four transformed results, and the average of the rotated results is taken to obtain a new comprehensive representation :

[0028]

[0029] in, Refers to rotating the image based on the center of the image to obtain multiple rotation angles. It means to average these rotated results.

[0030] The injected HH component adopts a weighted mixing method: the original sample x and the HH component of the specific pattern p are combined in proportion, and the proportion is determined by the hyperparameter Control, the combined image is controlled by composition:

[0031]

[0032] Use the inverse discrete wavelet transform IDWT to convert the wavelet domain samples back to the spatial domain to generate the initial watermark samples .

[0033] In combination with the first aspect, in some implementations of the first aspect, the method further includes: the DWT formula is as follows:

[0034]

[0035]

[0036]

[0037]

[0038] Among them, LL[x,y], HL[x,y], LH[x,y] and HH[x,y] represent the four subband coefficient matrices obtained by wavelet decomposition of the input image f[x,y], g[m] and g[n] are low-pass filters used to extract the low-frequency components of the signal, h[m] and h[n] are high-pass filters used to extract the high-frequency components of the image, m and n are the indexes of the filter, and convolution operation is performed on each point of the filter. 2x and 2y represent downsampling of each convolution result, and 2x-m and 2y-n represent translation operation on the input signal.

[0039] The sample rotation formula is as follows:

[0040]

[0041] in is the coordinate of the original pixel of the sample, is the coordinate of the pixel after rotation, Indicates the rotation angle;

[0042] The IDWT formula is as follows:

[0043]

[0044] Here, f[x,y] refers to the image generated by inverse wavelet transform.

[0045] In combination with the first aspect, in some implementations of the first aspect, the method further includes: calculating the cosine similarity between the initial watermark feature and the original feature as follows:

[0046]

[0047] Where I(∙) is the pre-trained feature extractor, and are the feature vectors extracted from the watermark sample and the original sample respectively, represents the norm of the eigenvector, ∙ represents the dot product of the vector;

[0048] After calculating the cosine similarity of all watermark samples and the corresponding original samples, construct the similarity list S:

[0049]

[0050] In the formula, Refers to the first watermark sample in the initial watermark sample set, refer to The corresponding clean sample, Refers to the initial watermark sample set The number of samples in Refers to the initial watermark sample set The last watermark sample in refer to The corresponding clean sample;

[0051] Sort the values ​​in the similarity list from high to low to get the sorted sample index list index:

[0052]

[0053] In the formula, It means sorting the elements in the list in descending order and returning the indices of the sorted elements.

[0054] Set the filter ratio , select the first one according to the sorted sample index list Proportional samples form a rough watermark sample set , The number of :

[0055]

[0056]

[0057] in, Refers to the total number of samples in the initial watermark sample set, Refers to the ratio of the rough trigger set to the initial watermark sample set;

[0058] For the rough watermark sample set , perform K-means clustering algorithm in feature space, divide the samples into K clusters, and get There are K feature clusters in total, which are solved by the following optimization problem:

[0059]

[0060] in, It is a cluster The cluster center represents the average feature vector of the samples in the cluster and is defined as follows:

[0061]

[0062] from The screening ratio is The number of watermark samples in the final watermark set is :

[0063] .

[0064] In combination with the first aspect, in some implementations of the first aspect, the method further includes: The number of watermark samples to be screened as follows:

[0065]

[0066] For each feature cluster Each sample in , calculate its distance to the cluster center :

[0067]

[0068] Distance Sort from large to small and select the ones that are far away from the cluster center. The previous sample, forming a cluster of watermark samples :

[0069]

[0070] Finally, the watermark sample sets selected from each cluster are aggregated to obtain the final watermark set :

[0071]

[0072] In the formula, It means merging K sets.

[0073] In combination with the first aspect, in some implementations of the first aspect, the method further includes: calculating the watermark extraction success rate as follows:

[0074]

[0075] Among them, n refers to the suspicious model In the collection The number of samples predicted correctly, Refers to the collection The number of samples.

[0076] Setting Thresholds If the suspicious model In the collection The accuracy is greater than the predetermined threshold , it is determined to be a pirated model, otherwise it is a non-pirated model. The formula is as follows:

[0077]

[0078] in, Indicates the probability of the event in brackets occurring.

[0079] In a second aspect, in order to achieve the above-mentioned purpose, the present invention discloses a personalized model copyright protection system based on DWT, comprising:

[0080] The sample conversion module is used to receive the clean sample set, input the original samples in the clean sample set into the pre-established generation network model, output the specific pattern samples, and perform domain conversion on the original samples and the specific pattern samples through discrete wavelet transform DWT and inverse discrete wavelet transform IDWT to finally obtain the initial watermark sample set;

[0081] A sample processing module is used to input the initial watermark samples and the original samples in the initial watermark sample set into a pre-trained feature extractor, output the cosine similarity between the initial watermark features and the original features, sort the initial watermark samples based on the cosine similarity to obtain a rough watermark sample set, and screen the rough watermark sample set to obtain a final watermark sample set;

[0082] The model judgment module is used to input the final watermark sample of the final watermark sample set into the pre-trained suspicious model, calculate the watermark extraction success rate, and if the watermark extraction success rate is greater than a set threshold, the suspicious model is judged to be a pirated model, otherwise it is judged to be a non-pirated model.

[0083] In another aspect of the present invention, in order to achieve the above-mentioned purpose, a terminal device is disclosed, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein the memory stores a computer program capable of running on the processor, and when the processor loads and executes the computer program, a personalized model copyright protection method based on DWT as described above is adopted.

[0084] Beneficial effects of the present invention:

[0085] The present invention significantly enhances the concealment of the watermark by perturbing the image in the wavelet domain rather than directly embedding the trigger pattern in the spatial domain. By using discrete wavelet transform to decompose and reconstruct the image, the generated watermark image is extremely natural visually and almost indistinguishable from the real image. The present invention generates a personalized trigger pattern based on the conditions of the input image and customizes a unique watermark for each sample, making it impossible for an attacker to crack or remove the watermark by generalization. The present invention designs a sample selection strategy that balances similarity and diversity, ensuring the naturalness and dispersion of the watermark sample relative to the original sample in the feature space, thereby effectively identifying suitable samples for embedding watermarks. BRIEF DESCRIPTION OF THE DRAWINGS

[0086] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0087] Figure 1 It is a schematic flow chart of the method of the present invention;

[0088] Figure 2 A schematic diagram of watermark sample generation in the present invention;

[0089] Figure 3 It is a schematic diagram of DWT in the present invention;

[0090] Figure 4 A schematic diagram of watermark sample screening in the present invention;

[0091] Figure 5 A schematic diagram of embedding a watermark in the present invention;

[0092] Figure 6 A schematic diagram of model ownership verification in the present invention;

[0093] Figure 7 Schematic diagram of the present invention's anti-pruning attack;

[0094] Figure 8 Schematic diagram of anti-NC detection of the present invention;

[0095] Fig. 9 It is a schematic diagram of the anti-Grad-cam detection of the present invention;

[0096] Fig.10 Schematic diagram of anti-STRIP detection of the present invention;

[0097] Fig.11 It is a schematic diagram of the system structure of the present invention. DETAILED DESCRIPTION

[0098] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0099] Embodiment 1:

[0100] The following is an introduction to the relevant terms involved in the embodiments of the present application:

[0101] Watermark: Image watermark uses the redundant data commonly found in image works to change the position or value of pixels (or transform domain coefficients) in the image's spatial domain (or transform domain) through a certain algorithm. Under the premise of not affecting the value and use of the original work, some copyright mark information (watermark) is embedded in the image work, and cannot be perceived by the human visual system. It can only be extracted through a dedicated detector or reader to achieve the purpose of protecting the copyright of the image work or verifying its integrity.

[0102] like Figure 1 As shown, a personalized model copyright protection method based on DWT, the method comprises the following steps:

[0103] S101: receiving a clean sample set, inputting the original samples in the clean sample set into a pre-established generative network model, outputting a specific pattern, performing domain conversion on the original samples and the specific pattern through discrete wavelet transform DWT and inverse discrete wavelet transform IDWT, and finally obtaining an initial watermark sample set;

[0104] like Figure 2As shown in the figure, for a batch of clean sample sets randomly selected from the training set, firstly, each sample in this batch of clean sample sets is input into the pre-established generative network model, and a specific pattern is output; then, each clean sample and its corresponding specific pattern are converted from the spatial domain to the wavelet domain by discrete wavelet transform DWT, and the corresponding four frequency domain components are obtained: LL, HL, LH, HH; secondly, the high-frequency area (HH) of the specific pattern is injected into the high-frequency area of ​​the original sample as the watermark trigger information; then, the inverse discrete wavelet transform IDWT is used to convert the watermarked sample back to the spatial domain to generate the watermark sample; finally, the target label is determined for the watermark sample to form the initial watermark sample set. The watermark sample generation process is described in detail:

[0105] This study performs a sample classification task, in which the original model Output C class probability vector: , original model The training set , , . Randomly select the original training set A small subset of is the clean sample set , The sample size is , Set to 0.2, generate watermark samples through the trigger function T(∙), and re-label them as the target class , get the initial watermark sample set .

[0106] The calculation process of the specific pattern sample is:

[0107] Each original sample in the clean sample set Input into the generative network model G to generate a personalized specific pattern p related to the input:

[0108] .

[0109] The generated patterns are highly variable, meaning that for any two different input samples, the corresponding output patterns will be different. and They are two different samples in the clean sample set, generating a network The corresponding output mode will be generated and To ensure the diversity of the output, the generator network must meet the following conditions:

[0110]

[0111] In order to quantify the difference between the output patterns of the generated network, Manhattan distance (L1 norm) can be introduced as a distance metric. and , and the corresponding output modes are and ,

[0112] Manhattan distance is defined as:

[0113]

[0114] Where n is the dimension of the generated network output, and Respectively indicate the output mode and The kth component of , the Manhattan distance We measured different samples and After generating two modes of network output and The sum of the absolute differences in all dimensions.

[0115] The difference based on Manhattan distance must satisfy the following formula:

[0116] such that ,in,

[0117] in, is a positive threshold indicating the minimum acceptable distance between output patterns, such that there exists a threshold that satisfies this formula.

[0118] The process of converting the domain of the original sample and the specific pattern sample by discrete wavelet transform DWT and inverse discrete wavelet transform IDWT is as follows:

[0119] Specifically, Figure 3 As shown in the figure, for a given image, it is first downsampled by passing through a vertical low-pass and high-pass filter, and then downsampled by passing through a horizontal low-pass and high-pass filter, and finally four sub-bands are obtained: LL, HL, LH, HH, which respectively represent approximate samples (which can also be understood as samples with certain information loss after compression), vertical edge features (extracted by vertical high-pass filters), horizontal edge features (extracted by horizontal high-pass filters), and diagonal features (both vertical and horizontal are processed by high-pass filters). The DWT formula is as follows:

[0120]

[0121]

[0122]

[0123]

[0124] Among them, LL[x,y], HL[x,y], LH[x,y] and HH[x,y] represent the four subband coefficient matrices obtained by wavelet decomposition of the input image f[x,y], g[m] and g[n] are low-pass filters used to extract the low-frequency components of the signal, h[m] and h[n] are high-pass filters used to extract the high-frequency components of the image, m and n are the indexes of the filter, and convolution operation is performed on each point of the filter. 2x and 2y represent downsampling of each convolution result, and 2x-m and 2y-n represent translation operation on the input signal.

[0125] The high-frequency information in the wavelet domain generated by the specific pattern p Rotate the center point clockwise by -5°, -10°, 5°, and 10° respectively, and average them to get a new , the sample rotation formula is as follows:

[0126]

[0127] in is the coordinate of the original pixel of the sample, is the coordinate of the pixel after rotation, Indicates the rotation angle;

[0128] Set a specific mode The high frequency region (HH) of In the high-frequency region, the HH component is injected in a weighted mixing manner: the original sample x and the HH component of the specific pattern p are combined in proportion, and the proportion is determined by the hyperparameter Control, the combined image is controlled by composition:

[0129]

[0130] Use the inverse discrete wavelet transform IDWT to convert the wavelet domain samples back to the spatial domain to generate the initial watermark samples .

[0131] The IDWT formula is as follows:

[0132]

[0133] Here, f[x,y] refers to the image generated by inverse wavelet transform.

[0134] For watermark sample Determine the target label , forming the initial watermark sample set .

[0135] S102: inputting the initial watermark samples and the original samples in the initial watermark sample set into a pre-trained feature extractor, outputting the cosine similarity between the initial watermark features and the original features, sorting the initial watermark samples based on the cosine similarity to obtain a rough watermark sample set, screening the rough watermark sample set to obtain a final watermark sample set;

[0136] like Figure 4 As shown in the figure, for the watermark samples in the initial watermark sample set and the clean samples in the corresponding clean sample set, they are first input into the pre-trained feature extractor, and the cosine similarity between each corresponding initial watermark sample feature and the clean sample feature is output; then similarity screening is performed, and the coarse watermark sample set is screened in descending order according to the cosine similarity; finally, diversity screening is performed, and the coarse watermark sample set is divided into K feature clusters according to the K-means clustering algorithm, and some watermark samples with a farther distance from the cluster center in each feature cluster are screened to obtain the final watermark sample set. The watermark sample screening process is described in detail:

[0137] For the initial watermark sample set Each watermark sample in , use the pre-trained feature extractor I(∙) to extract features from it and its corresponding original sample x:

[0138]

[0139] Next, calculate the cosine similarity between these two feature vectors , the formula is as follows:

[0140]

[0141] here, and are the feature vectors extracted from the watermark sample and the original sample respectively, represents the norm of the feature vector, and ∙ represents the dot product of the vector. Cosine similarity It measures the similarity of the two features. The closer the value is to 1, the more similar the two features are.

[0142] After calculating the cosine similarity of all watermark samples and the corresponding original samples, construct the similarity list S:

[0143]

[0144] In the formula, Refers to the first watermark sample in the initial watermark sample set, refer to The corresponding clean sample, Refers to the initial watermark sample set The number of samples in Refers to the initial watermark sample set The last watermark sample in refer to The corresponding clean sample.

[0145] Sort the values ​​in the similarity list from high to low to get the sorted sample index list index:

[0146]

[0147] In the formula, It means sorting the elements in the list in descending order and returning the indices of the sorted elements.

[0148] Set the filter ratio , select the first one according to the sorted sample index list Proportional samples form a rough watermark sample set , The number of :

[0149]

[0150]

[0151] in, Refers to the total number of samples in the initial watermark sample set, Refers to the ratio of the rough trigger set to the initial watermark sample set. is 0.5.

[0152] For the rough watermark sample set , perform K-means clustering algorithm in feature space, divide the samples into K clusters, and get There are K feature clusters in total. The goal of K-means is to minimize the square error of samples in each cluster, that is, to make the samples in the cluster as close to each other as possible, so that each cluster can represent a relatively dense feature area in the feature space. The following optimization problem is used to solve it:

[0153]

[0154] in, It is a cluster The cluster center represents the average feature vector of the samples in the cluster and is defined as follows:

[0155]

[0156] In order to ensure the diversity of samples in the feature cluster, it is necessary to The screening ratio is The number of watermark samples in the final watermark set is :

[0157]

[0158] Among them, setting is 0.2.

[0159] Since the sample density in each feature cluster is different, it is necessary to ensure that samples are evenly selected from each cluster to ensure comprehensive coverage of the feature space. , calculate the number of watermark samples to be screened as follows:

[0160]

[0161] By allocating the number of samples proportionally, the number of samples selected from each feature cluster is ensured to be consistent with its proportion in the overall sample set, thereby ensuring that the selected samples fully reflect the internal feature structure of the feature cluster.

[0162] In order to ensure that the final watermark sample set can fully cover different areas in the feature cluster, for each feature cluster , choose the number of clusters away from the center as Watermark sample.

[0163] First, for each feature cluster Each sample in , calculate its distance to the cluster center :

[0164]

[0165] Then, the samples are sorted according to the distance Sort from large to small and select the ones that are far away from the cluster center. The previous sample, forming a cluster of watermark sample sets :

[0166]

[0167] Finally, the watermark sample sets selected from each cluster are aggregated to obtain the final watermark set :

[0168]

[0169] In the formula, It means merging K sets.

[0170] S103: Input the final watermark sample of the final watermark sample set into the pre-trained suspicious model, calculate the watermark extraction success rate, and if the watermark extraction success rate is greater than a set threshold, determine the suspicious model as a pirated model, otherwise determine it as a non-pirated model.

[0171] Specifically, the present invention is further described below by way of embodiments: Figure 5 As shown in the figure, the correct normal label is set for the normal sample set, the wrong target label is set for the watermark sample set, and the two sample sets are mixed to train the watermark model. The process of watermark embedding is described in detail:

[0172] The first step is to design a normal model that requires the network to correctly identify normal samples. The normal input sample in is fed into the pre-trained model f without changing the label.

[0173] The second step is to design the watermark mode, which requires activating the watermark sample. The watermark sample in is input into the pre-trained model f and given the target label .

[0174] The third step is to reduce the impact of triggers on the classification accuracy of the model through classification loss Then, the loss is calculated for each input, and the final classification loss is The formula is as follows:

[0175]

[0176] In the formula, is the cross entropy loss function, It is the watermark embedding function, which means the watermark sample after adding the trigger to the original sample.

[0177] In the fourth step, in order to force the generated patterns to be non-reusable, that is, different samples generate different specific patterns, we need to use the difference loss Force regularization to train the generative network G, differential loss The formula is as follows:

[0178]

[0179] in, is a small value that prevents the denominator from reaching zero.

[0180] This regularization is a prerequisite for training the generative network G. Without it, the output of G will saturate to uniform firing.

[0181] In the fifth step, the final back propagation loss function is:

[0182]

[0183] The value of L is used as the loss value to back-propagate and update the model parameters, where is a hyperparameter that balances classification accuracy and discriminability loss.

[0184] Specifically, the present invention is further described below by way of embodiments: Figure 6 As shown, the watermark sample in the input watermark sample set queries the suspicious model and calculates the watermark extraction success rate Acc. If the watermark extraction success rate Acc of the suspicious model on the watermark sample set is greater than the predefined threshold , then the model owner can determine that the model is a pirated model, otherwise it is not a pirated model. The ownership verification process is described in detail:

[0185] Input watermark sample set The watermark sample in the query suspicious model , assuming The total number of samples in is , where there are n samples with output labels and target labels The watermark extraction success rate is calculated by the following formula:

[0186]

[0187] Among them, n refers to the suspicious model In the collection The number of samples predicted correctly, Refers to the collection The number of samples.

[0188] Setting Thresholds is 95%, if the suspicious model is in the set The accuracy is greater than the predetermined threshold , then the model owner can determine that the model is a pirated model, otherwise it is not a pirated model. The formula is as follows:

[0189]

[0190] in, Represents the probability of the event in the brackets occurring, that is, the probability that the accuracy of the model exceeds the threshold.

[0191] We evaluate the performance of our method on two datasets: MNIST and CIFAR-10. In our experiments, the number of feature clusters K is set to 6, and the hyperparameter of the high-frequency information embedding ratio is Set to 0.2, a hyperparameter that balances classification accuracy and discriminability loss The SGD optimizer is used to train the classifier network with a learning rate of 0.001, and the Adam optimizer is used to train the generator network with a learning rate of 0.001.

[0192] Fidelity: The purpose of fidelity is to test whether the performance of the model will be reduced due to watermark embedding when tested on clean samples. We use the specified network architecture in the above dataset to train a watermark-free DNN model, and then use the current mainstream watermarking methods (Abstract, Blend, Entangled, and LSB) to compare with our method. The accuracy results are shown in Table 1. According to the data in the second and fourth columns of the table, the clean test accuracy of this method on the MNIST dataset is 99.50%, and the clean test accuracy on the CIFAR-10 dataset is 94.13%, which is similar to or even slightly better than the current mainstream methods.

[0193] In addition, sample fidelity means that the perturbed samples should be visually close to the original samples. To quantify the visual quality, we determine the pixel mean square error (MSE), average peak signal-to-noise ratio (PSNR, dB), and structural similarity (SSIM) between the original samples and the corresponding watermarked samples in the test set. The experimental results on CIFAR-10 are shown in Table 2. The pixel mean square error of this method is 6.56, the average PSNR is 34.68dB, and the average SSIM is 0.9919. All visual indicators are better than the current mainstream watermarking methods (Abstract, Blend, Entangled, and LSB). This shows that the watermarked samples have good visual quality and can be guaranteed to be imperceptible.

[0194] Effectiveness: Effectiveness refers to the classification accuracy of the watermark model when using trigger query. We use the watermark sample with added trigger to test, and the results are shown in Table 1. According to the data in columns 3 and 5 of the table, the method in this study can achieve very high results. When using trigger sample query, the watermark accuracy on the MNIST dataset is 99.83%, and the watermark accuracy on the CIFAR-10 dataset is 99.60%, which is better than the current mainstream watermark method.

[0195] Robustness: Robustness evaluates the ability to reconstruct the embedded watermark from the labeled DNN model when the watermarking system is attacked by an adversary. Figure 7The correlation between the number of pruned neurons and the test accuracy and watermark accuracy under pruning attacks is shown. It can be observed that the test accuracy always decreases earlier than the watermark accuracy. Our watermark model can still achieve a watermark accuracy close to 100% when the number of pruned channels reaches 60 for the MNIST dataset and 500 for the CIFAR-10 dataset.

[0196] Detection resistance: Detection resistance evaluates the resistance of the watermark model when it is detected by an adversary targeting the trigger. Figure 8 The performance of the proposed method in NC detection is shown, with an anomaly index of 1.51 on the MNIST dataset and 1.25 on the CIFAR-10 dataset, which are always below the threshold of 2. Fig. 9 It shows that for Grad-cam detection, the heatmap of the watermarked sample with added triggers is almost the same as the clean sample. Fig.10 It is shown that the entropy distribution produced by this method is similar to that of clean samples and cannot be distinguished by the STRIP detection method.

[0197] Table 1: Test accuracy and watermark accuracy of different datasets after watermarking

[0198]

[0199] Table 2: Visual effects of CIFAR-10 after watermarking (MSE, PSNR and SSIM)

[0200]

[0201] Embodiment 2: In the second aspect, as Fig.11 As shown, in order to achieve the above-mentioned purpose, the present invention discloses a personalized model copyright protection system based on DWT, comprising:

[0202] The sample conversion module 11 is used to receive a clean sample set, input the original samples in the clean sample set into a pre-established generation network model, output a specific pattern sample, and perform domain conversion on the original samples and the specific pattern samples through discrete wavelet transform DWT and inverse discrete wavelet transform IDWT to finally obtain an initial watermark sample set;

[0203] The sample processing module 12 is used to input the initial watermark samples and the original samples in the initial watermark sample set into a pre-trained feature extractor, output the cosine similarity between the initial watermark features and the original features, sort the initial watermark samples based on the cosine similarity to obtain a rough watermark sample set, and screen the rough watermark sample set to obtain a final watermark sample set;

[0204] The model determination module 13 is used to input the final watermark sample of the final watermark sample set into the pre-trained suspicious model, calculate the watermark extraction success rate, and if the watermark extraction success rate is greater than a set threshold, the suspicious model is determined to be a pirated model, otherwise it is determined to be a non-pirated model.

[0205] Based on the same inventive concept, the present invention also provides a computer device, which includes: one or more processors, and a memory for storing one or more computer programs; the program includes program instructions, and the processor is used to execute the program instructions stored in the memory. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is used to implement one or more instructions, specifically for loading and executing one or more instructions in a computer storage medium to implement the above method.

[0206] It should be further explained that, based on the same inventive concept, the present invention also provides a computer storage medium, on which a computer program is stored, and the computer program is executed by the processor to execute the above method. The storage medium can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electrical, magnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0207] In the description of this specification, the description with reference to the terms "one embodiment", "example", "specific example", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0208] The above shows and describes the basic principles, main features and advantages of the present disclosure. Those skilled in the art should understand that the present disclosure is not limited by the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present disclosure. Without departing from the spirit and scope of the present disclosure, the present disclosure may have various changes and improvements, and these changes and improvements fall within the scope of the present disclosure to be protected.

Claims

1. A personalized model copyright protection method based on DWT, characterized in that: The method comprises the following steps: Receive a clean sample set, input the original samples in the clean sample set into the pre-established generative network model, output a specific pattern sample, transform the original samples and the specific pattern samples by discrete wavelet transform DWT and inverse discrete wavelet transform IDWT, and embed watermark information in the wavelet domain, finally obtaining an initial watermark sample set; The process of converting the domain of the original sample and the specific pattern sample by discrete wavelet transform DWT and inverse discrete wavelet transform IDWT is as follows: Use discrete wavelet transform DWT to transform the original sample x and specific pattern p from the spatial domain to the wavelet domain: DWT(x)→(x LL ,x HL ,x LH ,x HH } DWT(p)→{p LL ,p HL ,p LH ,p HH } Among them, x LL , x HL , x LH , x HH They refer to the four frequency subbands into which the original sample x is decomposed by DWT, representing the low-frequency part, the horizontal high-frequency part, the vertical high-frequency part, and the diagonal high-frequency part respectively; p LL , p HL , p LH , p HH They refer to the four frequency subbands into which a specific mode p is decomposed by DWT, representing the low-frequency part, the horizontal high-frequency part, the vertical high-frequency part, and the diagonal high-frequency part respectively; The high-frequency region HH of the specific pattern p is injected into the high-frequency region of the original sample x as a watermark trigger; Before injection, the high-frequency region HH of a specific mode p is fine-tuned: the HH region is rotated at different angles around the center point to generate four transformed results, and the average of the rotated results is taken to obtain a new comprehensive representation Here, rotate(·) means rotating the image based on the center of the image to obtain multiple rotation angles, and mean(·) means averaging the results after rotation. The injection of HH components adopts a weighted mixing method: the original sample x and the HH components of the specific pattern p are combined in proportion, and the proportion is controlled by the hyperparameter α. The combined image is represented by {x LL , x HL , x LH , z HH }composition: Use the inverse discrete wavelet transform IDWT to convert the wavelet domain samples back to the spatial domain to generate the initial watermark sample x p =IDWT({x LL , x HL , x LH , z HH }); The DWT formula is as follows: Among them, LL[x,y], HL[x,y], LH[x,y] and HH[x,y] represent the four subband coefficient matrices obtained by wavelet decomposition of the input image f[x,y], g[m] and g[n] are low-pass filters used to extract the low-frequency components of the signal, h[m] and h[n] are high-pass filters used to extract the high-frequency components of the image, m and n are the indexes of the filter, and the convolution operation is performed on each point of the filter. 2x and 2y represent downsampling of each convolution result, and 2x-m and 2y-n represent translation operations on the input signal; The sample rotation formula is as follows: Where (x0, y0) is the coordinate of the original pixel of the sample, (x, y) is the coordinate of the pixel after rotation, and θ represents the rotation angle; The IDWT formula is as follows: Where f[x,y] refers to the image generated by inverse wavelet transform; is the watermark sample x p Determine the target label y t , forming the initial watermark sample set The initial watermark samples and the original samples in the initial watermark sample set are input into the pre-trained feature extractor, and the cosine similarity between the initial watermark features and the original features is output. The initial watermark samples are sorted based on the cosine similarity to obtain a rough watermark sample set. The rough watermark sample set is screened for diversity using the K-means clustering algorithm to obtain the final watermark sample set. The final watermark sample of the final watermark sample set is input into the pre-trained suspicious model, and the watermark extraction success rate is calculated. If the watermark extraction success rate is greater than the set threshold, the suspicious model is determined to be a pirated model, otherwise it is determined to be a non-pirated model.

2. The DWT-based personalized model copyright protection method according to claim 1, characterized in that: The calculation process of the specific pattern sample is: Each original sample x in the clean sample set is input into the generative network model G to generate a personalized specific pattern p related to the input: p=G(x).

3. The DWT-based personalized model copyright protection method according to claim 2, characterized in that: The generated network model G satisfies the following conditions: Among them, x is the original sample, x i and x j are two different samples in the clean sample set, and the Manhattan distance L1 norm is introduced as the distance metric. i and x j , the corresponding output modes are C(x i )=p i and G(x j )=p j , Manhattan distance is defined as: Where n is the dimension of the generated network output, and Respectively represent the output mode G(x i ) and G(x j )’s kth component, Manhattan distance d man (G(x i ), G(x j )) measures different samples x i and x j After generating the two patterns G(x i ) and G(x j ) is the sum of the absolute differences in all dimensions; The difference based on Manhattan distance must satisfy the following formula: in, Where ε is a positive threshold representing the minimum acceptable distance between output patterns, such that there exists a threshold that satisfies the formula.

4. The DWT-based personalized model copyright protection method according to claim 1, characterized in that: The cosine similarity between the initial watermark feature and the original feature is calculated as follows: In the formula, S(x p , x) is the cosine similarity between the two vectors, I(·) is the pre-trained feature extractor, I(x p ) and I(x) are the feature vectors extracted from the watermark sample and the original sample respectively, ||I(x p )|| represents the norm of the eigenvector, · represents the dot product of the vector; After calculating the cosine similarity of all watermark samples and the corresponding original samples, construct the similarity list S: S={S(x p1 ,x1),S(x p2 ,x2),…,S(x pN*ρ ,x N*ρ )} In the formula, x p1 Refers to the first watermark sample in the initial watermark sample set, x1 refers to x pl The corresponding clean sample, N*ρ refers to the initial watermark sample set D ori The number of samples in x pN*ρ Refers to the initial watermark sample set D ori The last watermark sample in x N*ρ Point to x pN*ρ The corresponding clean sample; Sort the values ​​in the similarity list from high to low to get the sorted sample index list index: index = argsort(S, desending) In the formula, argsort(·, desending) means sorting the elements in the list in descending order and returning the index of the sorted elements; Set the screening ratio ρ1, select the first ρ1 proportion of samples according to the sorted sample index list, and form a rough watermark sample set D rough , D rough The number is N rough : A rough =N s *ρ1 D rough ={x i |i∈index[1:N rough ]} Among them, N s Refers to the total number of samples in the initial watermark sample set, ρ1 refers to the proportion of the coarse trigger set to the initial watermark sample set; For the rough watermark sample set D rough , perform K-means clustering algorithm in the feature space, divide the samples into K clusters, and get C1, C2, C3, …, C K There are K feature clusters in total, which are solved by the following optimization problem: Among them, c k It is cluster C k The cluster center represents the average feature vector of the samples in the cluster and is defined as follows: From D rough The watermark samples with a screening ratio of ρ2 are selected, and the final number of watermark sets is N final : N final =N rough *ρ2。 5. The DWT-based personalized model copyright protection method according to claim 4, characterized in that: The feature cluster C k The number of watermark samples to be screened N k as follows: For each feature cluster C k For each sample x in i , calculate its distance d(x i , C k ): d(x i ,C k )=||I(x i )-c k || The samples are divided into i , C k ) Sort from large to small and select N that are farthest from the cluster center. k The previous sample, forming a cluster of watermark samples Finally, the watermark sample sets selected from each cluster are aggregated to obtain the final watermark set D final : In the formula, It means merging K sets.

6. The DWT-based personalized model copyright protection method according to claim 1, characterized in that: The watermark extraction success rate is calculated as follows: Where n refers to the number of suspicious models f′ in the set D final The number of samples predicted correctly, N fjnal It refers to the set D final The number of samples; Set the threshold τ, if the suspicious model is in the set D final If the accuracy is greater than the predetermined threshold τ, it is determined to be a pirated model, otherwise it is a non-pirated model. The formula is as follows: Pr{Acc(D final ,f′)>τ}=1 Where Pr{·} represents the probability of the event in brackets occurring.

7. A DWT-based personalized model copyright protection system, which adopts a DWT-based personalized model copyright protection method according to any one of claims 1 to 6, characterized in that: include: The sample conversion module is used to receive the clean sample set, input the original samples in the clean sample set into the pre-established generation network model, output the specific pattern samples, and perform domain conversion on the original samples and the specific pattern samples through discrete wavelet transform DWT and inverse discrete wavelet transform IDWT to finally obtain the initial watermark sample set; A sample processing module is used to input the initial watermark samples and the original samples in the initial watermark sample set into a pre-trained feature extractor, output the cosine similarity between the initial watermark features and the original features, sort the initial watermark samples based on the cosine similarity to obtain a rough watermark sample set, and screen the rough watermark sample set to obtain a final watermark sample set; The model judgment module is used to input the final watermark sample of the final watermark sample set into the pre-trained suspicious model, calculate the watermark extraction success rate, and if the watermark extraction success rate is greater than a set threshold, the suspicious model is judged to be a pirated model, otherwise it is judged to be a non-pirated model.

8. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that: The memory stores a computer program that can be run on the processor. When the processor loads and executes the computer program, a personalized model copyright protection method based on DWT as described in any one of claims 1 to 6 is adopted.

Images