Model fingerprint intellectual property protection method based on frequency visual angle

Abstract

The invention discloses a model fingerprint intellectual property protection method based on a frequency visual angle, and the method comprises the steps: extracting the frequency domain information of a model weight: S1, extracting a 3D array of a convolution kernel weight of a bottom convolution layer of a model, and obtaining a frequency domain coefficient through 3D discrete cosine transform (3DDCT); s2, the model owner extracts low-intermediate frequency components in the frequency domain coefficient according to a preset threshold value, and orderly connects the low-intermediate frequency components into a feature vector; s3, using a random projection technology to bind the feature vector and the identity information to generate a fingerprint, and submitting the fingerprint to a trusted third party (TTP) for timestamp verification; and S4, performing suspicious model detection: repeating the above steps on the suspicious model to obtain a feature vector, performing calculation by using an identity matrix of a model owner to obtain a fingerprint of the suspicious model, comparing the fingerprint with a fingerprint in a TTP database, and judging whether the suspicious model is pirated or not through cosine similarity. According to the method, the intellectual property protection capability of the model is effectively improved, and relatively high robustness is shown.

Description

Technical Field

[0001] The present invention relates to the field of model copyright protection, and in particular to a method for protecting model fingerprint intellectual property rights based on a frequency perspective. Background Art

[0002] Training deep neural networks is resource-intensive and expensive, typically requiring significant investments in high-quality data, expert knowledge, and computing power. Consequently, these valuable models are increasingly becoming targets for thieves. Previous research has shown that attackers can obtain exact replicas of original models through model-sharing platforms or even steal them through model extraction attacks. Therefore, protecting the intellectual property rights of models from infringement is crucial.

[0003] Existing methods for protecting model intellectual property rights mainly include model watermarking and fingerprint recognition technology, which enable model owners to verify and assert their ownership in the event of model theft, thereby providing a means to resolve intellectual property disputes.

[0004] Watermarking methods embed a watermark representing ownership into the model during training and then verify ownership by extracting the watermark from the suspicious model. Specifically, feature-based white-box watermarking methods embed ownership identifiers into the model weights by modifying the loss function during training. In contrast, backdoor-based black-box watermarking methods insert specific triggers during training, allowing the owner to verify the pirated model based on different model outputs. However, the retraining process of embedding watermarks is costly, and the embedded watermarks may lead to a loss in model accuracy. In addition, while some watermarking methods have demonstrated robustness against various fuzzing attacks (such as pruning or fine-tuning), recent studies have shown that model watermarks are still subject to the threats of tampering and watermark overwriting.

[0005] In contrast, DNN fingerprinting methods have garnered increasing attention due to their non-invasive nature. They prove model ownership by extracting unique features from existing models without modifying them. Specifically, the decision boundary can reflect the distinctive characteristics of a model, and various methods have been proposed to extract fingerprints from the decision boundary, including leveraging feature points close to the decision boundary or adversarial examples. Previous research has evaluated the robustness of fingerprinting to attacks that obfuscate existing ownership. However, recent research has shown that modifying a model's high-frequency coefficients can shift the model's decision boundary and weights without significantly affecting its performance. This boundary shift renders boundary-based fingerprinting methods ineffective.

[0006] This paper proposes a frequency-based model fingerprinting method for intellectual property protection. By extracting frequency-domain features rather than decision boundary features, it effectively defends against fuzzy attacks based on high-frequency coefficient modification and demonstrates robustness against existing common fuzzy attacks. This method only requires frequency transformation of the model, without the need for retraining or adversarial training. Summary of the Invention

[0007] The technical problem to be solved by the present invention is: in response to the technical problems existing in the prior art, the present invention designs a model fingerprint ownership protection method based on a frequency perspective. The model owner only needs to perform frequency transformation on the model and upload the calculated fingerprint to a trusted third party.

[0008] To solve the above technical problems, the present invention proposes a technical solution: a method for fuzzy attack and fingerprint defense in model fingerprint intellectual property protection based on a frequency perspective, comprising the following steps:

[0009] A frequency-based model fingerprint intellectual property protection method, characterized by comprising:

[0010] S1, the model owner extracts the 3D array corresponding to each convolution kernel from the bottom convolution layer of the model, and uses 3DDCT to perform frequency domain transformation on the 3D array to obtain frequency domain coefficients;

[0011] S2, the model owner extracts the low-medium frequency part of each underlying convolution kernel DCT frequency domain coefficient according to a preset threshold and coefficient index, and concatenates the low-medium frequency parts into a feature vector;

[0012] S3: The model owner uses the model owner's identity information and a pseudo-random number generator to generate a unique random Bernoulli matrix. The model owner then uses random projection technology to bind the eigenvector to the matrix to generate a fingerprint. The model owner then submits the fingerprint and identity information to a trusted third party (TTP) for timestamp verification to ensure the validity of the fingerprint.

[0013] S4. When the model owner suspects that a model is a pirated model, the feature vector of the suspicious model is extracted according to steps S1 and S2, and the fingerprint of the suspicious model is calculated using the identity matrix of the model owner. The fingerprint is compared with the fingerprint registered in the TTP database by cosine similarity. The larger the cosine similarity value, the more similar the model is and the more likely it is a pirated model.

[0014] Furthermore, the model frequency domain information extraction method described in step S1 includes: performing 3D DCT on each convolution kernel in the bottom convolution layer of the model (usually the first layer) to obtain frequency domain coefficients.

[0015] Furthermore, the low-medium frequency coefficient extraction method described in step S2 includes: sorting the frequency domain coefficients from low to high by the index and sum of the frequency domain coefficients. When the coefficient index and sum are less than the set threshold, the frequency domain coefficient is identified as a low-medium frequency component, and all low-medium frequency components that meet the conditions in each convolution kernel are extracted in order according to the above steps.

[0016] Furthermore, the feature vector composition method described in step S2 includes: connecting the extracted low- and medium-frequency components of each convolution kernel in the order of finding the convolution kernel into a one-dimensional feature vector, and the feature vector is used to characterize the frequency domain information of the model and serve as the basis for generating the model fingerprint.

[0017] Furthermore, the fingerprint generation method described in step S3 includes: the model owner uses their identity information and a pseudo-random number generator to generate a unique random Bernoulli matrix. The model's eigenvectors are projected into the random space defined by this matrix to obtain the model's frequency-domain fingerprint. The model owner then submits this fingerprint and identity matrix to the TTP for timestamp verification to ensure the fingerprint's validity.

[0018] Furthermore, the method for determining whether a model is pirated as described in step S4 includes: extracting low-medium frequency feature vectors from the suspicious model, using the model owner identity matrix in the TTP database to calculate the fingerprint of the suspicious model, and comparing the fingerprint with the fingerprint registered in the TTP database by cosine similarity. If the cosine similarity exceeds a preset threshold, it is determined to be a pirated model.

[0019] Furthermore, the model fingerprint generation method described in steps S1-S3 is applicable to the intellectual property protection of any deep learning model, machine learning model or other model based on weight parameters.

[0020] Compared with the prior art, the advantages of the present invention are:

[0021] 1. This invention is the first fingerprint defense method based on the frequency perspective in model intellectual property protection, which can effectively protect the ownership of the model. Compared with existing watermarking schemes, this invention does not require fine-tuning of model weights to embed watermarks, thereby greatly improving the model training efficiency. Compared with existing fingerprinting schemes, this invention can resist fuzzy attacks based on high-frequency coefficient modification.

[0022] 2. The present invention improves the efficiency of fingerprint generation to a great extent by extracting frequency domain features rather than decision boundary features.

[0023] 3. The present invention uses TTP to perform timestamp verification, ensuring the validity and non-tamperability of fingerprints, and providing a reliable solution to intellectual property disputes.

[0024] 4. The present invention is applicable to the intellectual property protection of deep learning models, machine learning models or other models based on weight parameters and has universal applicability. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 Schematic diagram of a specific embodiment of the present invention.

[0026] Figure 2 These are the robustness test results of a specific embodiment of the present invention under model pruning attacks, fine-tuning attacks, and high-frequency coefficient tampering attacks on different data sets.

[0027] Figure 3 These are the robustness test results of the specific embodiments of the present invention on different models under model pruning attacks, fine-tuning attacks, and high-frequency coefficient tampering attacks.

[0028] Figure 4 The following is a comparison of the tracing success rate and embedding efficiency between the specific embodiments of the present invention and existing solutions (IPGuard, Deepfool, and FUAP).

[0029] Figure 5 This is the test result of the innocent model according to the specific embodiment of the present invention. DETAILED DESCRIPTION

[0030] The present invention will be further described below in conjunction with the accompanying drawings and specific preferred embodiments, but the scope of protection of the present invention is not limited thereby.

[0031] like Figure 1 As shown, the frequency-based model fingerprint intellectual property protection framework of this embodiment includes model owners and suspicious models. The model owner is responsible for training the model and can distribute the model by sharing model parameters or providing APIs. The goal of the model owner is to protect the intellectual property rights of the model through model ownership protection technology. Specifically, after extracting the fingerprint from the model, the owner can verify the ownership by extracting the fingerprint from the suspicious model if unauthorized use is detected. The suspicious model may be a pirated model or an innocent model. The pirated model refers to the illegal use of the knowledge and capabilities of the model owner's model to obtain a model with good performance at a relatively low cost. The innocent model is an innocent model that is independently trained from scratch and has nothing to do with the model owner's model.

[0032] In this implementation, the model owner mainly undertakes two tasks: one is to generate and register fingerprints. After the model training is completed, the model frequency domain features are extracted, the fingerprint is generated and calculated, and the fingerprint is registered to ensure the non-repudiation and irrevocability of the fingerprint.

[0033] In the frequency domain feature extraction stage of the model, the low-level convolution kernel weights are selected because of their universality and stability. The distribution of these weights is also more difficult to change, making them more suitable for feature extraction. According to the following formula (1), 3D DCT is performed on each convolution kernel in each convolution layer f(x, y, z) to obtain the frequency domain coefficients F(u, v, w), where n x , n y , n z Indicates the number of input channels, the height and width of the convolution kernel.

[0034]

[0035] After obtaining the frequency domain coefficients, since the model gives priority to the low-frequency components of the training data during the entire training process, it is more sensitive to low-frequency channels than high-frequency channels, and gradually proceeds from low frequency to high frequency when approximating the target function, and the energy in the model weights is mainly concentrated in the low-frequency DCT components, the low-intermediate frequency components can be extracted as the feature vectors of the model. Since low-frequency components are usually located at smaller index positions, while high-frequency components correspond to larger index positions. Therefore, for each frequency domain coefficient F i (u,v,w), where u, v, w represent the index of the frequency domain coefficients, and u∈[0,n u -1], v∈[0,n v -1], w∈[0,n w -1],n u 、n v 、n w Represent the number of input channels, the height and width of the filter respectively. According to the preset threshold ratio, it is judged whether the index and u+v+w are less than the threshold ratio. If the conditions are met, the frequency domain coefficient F(u,v,w) is identified as the low-IF component. All low-IF components that meet the conditions are extracted and these low-IF components are sequentially connected into a one-dimensional feature vector V c .

[0036] During the fingerprint generation phase, the model owner uses a pseudo-random number generator and the model owner’s identity information (e.g., ID number or corporate entity number) to create a unique random Bernoulli matrix t0 represents the eigenvector V c The length of t corresponds to the length of the final generated fingerprint f. Note that the matrix U needs to be normalized to generate a random projection matrix P, ensuring that each column of P is a unit vector. Finally, as shown in formula (2), the eigenvector V of the model is c Project it onto the random space defined by the matrix P to obtain the fingerprint f.

[0037]

[0038] During the fingerprint registration phase, the generated fingerprint f and identity information U are submitted to the TTP for timestamp authentication and storage to ensure the validity of the fingerprint.

[0039] The other is the task of detecting pirated models. When the ownership of a suspicious model needs to be verified, the model owner needs to obtain the low-intermediate frequency coefficients corresponding to the same position weights from the suspicious model to form a feature vector V c ′, then use the same random matrix U and random projection method to extract the eigenvector V from the suspect model c ', and the calculated fingerprint is compared with the fingerprint registered in the TTP database. In the comparison process, this method uses cosine similarity as a metric, and its calculation method is shown in formula (3):

[0040]

[0041] In this process, fingerprint vectors f and f′ represent the feature representations of the victim model and suspect model, respectively. A higher cosine similarity value indicates a higher match between the two fingerprints, and thus a greater similarity between the two models. If the calculated cosine similarity exceeds a preset threshold, and the timestamp in the TTP database indicates that the original model was registered earlier than any of the allegedly registered suspect models, the suspect model can be confirmed to belong to the original author.

[0042] In this embodiment, the technical solution is verified through simulation experiments, and PyTorch is used for model training. The simulation experiments are carried out on a physical machine containing an Intel i5-13600K CPU and an NVIDIA GeForce RTX 4070 graphics card.

[0043] In the simulation experiment, the pirated model was obtained by applying five fuzzy attack methods to the original model: weight pruning, convolution kernel pruning (p and q are both pruning rates), retraining fine-tuning, transfer learning fine-tuning, and high-frequency coefficient modification; the innocent model was obtained by independent training using the same dataset or model structure as the original model, or the same dataset and model but different initialization conditions.

[0044] In simulation experiments, in order to verify the applicability of the frequency-based model fingerprint intellectual property protection method on different models and datasets, the AlexNet model was tested on four popular image classification datasets (MNIST, Fashion MNIST [FMNIST], CIFAR-10, and CIFAR-100), and the robustness of four model architectures (CNN model containing two convolutional layers, one maximum pooling layer, and two fully connected layers, as well as AlexNet, ResNet18, and VGG16) were tested on the CIFAR-10 dataset. The test results are shown in the figure below. Figure 2 、 3 The results show that the frequency-based model fingerprinting IP protection method consistently achieves over 97% similarity across all attack scenarios. Specifically, in the convolution kernel pruning attack scenario, even when some convolution kernels are removed, resulting in partial loss of fingerprint information, the proposed method still maintains a fingerprint recognition rate of 97.72% to 99.62%. This demonstrates that pruning the model weights does not destroy the frequency-based fingerprint; even with the loss of some convolution kernel information, the remaining fingerprint still provides sufficient recognition accuracy. Furthermore, as model complexity increases, the magnitude of the change in model performance and fingerprint similarity decreases under the same attack conditions. This can be attributed to differences in model structural complexity. Larger models typically exhibit higher parameter redundancy, meaning a relatively lower proportion of effective parameters. Due to the presence of redundant parameters, the impact of attacks on model performance is more dispersed, making the model's key features and fingerprint similarity more stable under attack.

[0045] In simulation experiments, in order to verify the effectiveness of the frequency-based model fingerprint IP protection method, it is compared with four advanced model IP methods: (a) IPGuard finds points close to the decision boundary by optimizing the objective function; (b) Deepfool uses geometric properties to find the minimum perturbation to make the input point cross the decision boundary; (c) FUAP uses universal adversarial perturbation to represent the decision boundary. Figure 4As shown in the figure, when faced with different types of fuzzy attacks, the frequency-based model fingerprint intellectual property protection method can maintain a model accuracy (ACC) of 85.6% to 89.34%, while the fingerprint recognition rate is almost maintained at 100%. However, when encountering frequency-based fuzzy attacks, the fingerprint recognition rates of IPGuard, DeepFool, and UAPS drop significantly. Since this method uses low-frequency features as fingerprints, the modification of high-frequency coefficients does not affect the effectiveness of the fingerprint. In contrast, it can be observed that as the degree of modification of high-frequency coefficients increases, the fingerprint recognition rate of the baseline method gradually decreases. This is because most high-frequency coefficients are initially close to zero, and as the modified values further deviate from zero, the model weights and decision boundaries will undergo greater changes, resulting in a significant decrease in the fingerprint recognition rate of the baseline method.

[0046] In the simulation experiment, in order to verify the fingerprint uniqueness of the model fingerprint intellectual property protection method based on the frequency perspective, the fingerprint similarity FS between models trained independently on the same dataset or trained with the same model architecture was compared. Figure 5 As shown, the fingerprint similarity between independently trained models is generally low and well below the threshold used to identify pirated models, indicating that the fingerprints are unique. Notably, even when trained on the same dataset and using the same model architecture, these models still exhibit low fingerprint similarity. This is due to the non-convex nature of the neural network loss function, meaning that different initializations lead to different local minima for the model parameters. Therefore, when two independent entities train their own models using the same data and model architecture, these models should be considered independent entities.

[0047] The above description is merely a preferred embodiment of the present invention and does not constitute any form of limitation to the present invention. Although the present invention has been disclosed above based on preferred embodiments, it is not intended to limit the present invention. Therefore, any simple modifications, equivalent variations, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention shall fall within the scope of protection of the technical solution of the present invention.

Claims

1. A model fingerprint intellectual property protection method based on frequency perspective. Characterized by: include: S1, the model owner extracts the 3D array corresponding to each convolution kernel from the bottom convolution layer of the model, and uses three-dimensional discrete cosine transform (3D DCT) to perform frequency domain transformation on the 3D array to obtain frequency domain coefficients; S2, the model owner extracts the low-medium frequency part of each underlying convolution kernel DCT frequency domain coefficient according to a preset threshold and coefficient index, and concatenates the low-medium frequency parts into a feature vector; S3: The model owner uses their identity information and a pseudo-random number generator to generate a unique random Bernoulli matrix. They then use random projection technology to bind the eigenvector to the matrix to generate a fingerprint. The fingerprint and identity information are then submitted to a trusted third party (TTP) for timestamp verification to ensure the validity of the fingerprint. S4. When the model owner suspects that a model is a pirated model, the feature vector of the suspicious model is extracted according to steps S1 and S2, and the fingerprint of the suspicious model is calculated using the identity matrix of the model owner. The fingerprint is compared with the fingerprint registered in the TTP database by cosine similarity. The larger the cosine similarity value, the more similar the model is and the more likely it is a pirated model.

2. The frequency-based model fingerprint intellectual property protection method according to claim 1 is characterized in that: The model frequency domain information extraction method described in step S1 includes: performing 3D DCT on each convolution kernel in the bottom convolution layer of the model (usually the first layer) to obtain frequency domain coefficients.

3. The frequency-based model fingerprint intellectual property protection method according to claim 1 is characterized in that: The low-medium frequency coefficient extraction method described in step S2 includes: sorting from low frequency to high frequency by the index of the frequency domain coefficient. When the coefficient index sum is less than the set threshold, the frequency domain coefficient is identified as a low-medium frequency component, and all low-medium frequency components that meet the conditions in each convolution kernel are extracted in order according to the above steps.

4. The frequency-based model fingerprint intellectual property protection method according to claim 1 is characterized in that: The feature vector composition method described in step S2 includes: connecting the extracted low-medium frequency components of each convolution kernel in sequence according to the convolution kernel order into a one-dimensional feature vector, and the feature vector is used to characterize the frequency domain information of the model and serve as the basis for generating the model fingerprint.

5. The frequency-based model fingerprint intellectual property protection method according to claim 1 is characterized in that: The fingerprint generation method described in step S3 involves the model owner using their identity information and a pseudorandom number generator to generate a unique random Bernoulli matrix. The model's eigenvectors are projected into the random space defined by this matrix to obtain the model's frequency-domain fingerprint. The model owner then submits this fingerprint and identity matrix to the TTP for timestamp verification to ensure the fingerprint's validity.

6. The frequency-based model fingerprint intellectual property protection method according to claim 1 is characterized in that: The method for determining whether a model is pirated as described in step S4 includes: extracting low-medium frequency feature vectors from the suspicious model, using the model owner identity matrix in the TTP database to calculate the fingerprint of the suspicious model, and comparing the fingerprint with the fingerprint registered in the TTP database by cosine similarity. If the cosine similarity exceeds a preset threshold, it is determined to be a pirated model.

7. The frequency-based model fingerprint intellectual property protection method according to claim 1 is characterized in that: The model fingerprint generation method described in steps S1-S3 is applicable to the intellectual property protection of any deep learning model, machine learning model, or other model based on weight parameters.

Images