Active protection method and system for deep learning model driven by frequency domain trigger

Abstract

The invention discloses a deep learning model active protection method driven by a frequency domain trigger. The method comprises the following steps: firstly, obtaining a white box access permission of a to-be-protected target model; obtaining a model data set and generating a frequency domain trigger; embedding the generated frequency domain trigger into a data set; training an active protection model, updating weight parameters in the model, and enabling the model to depend on a trigger data set; and finally, inputting the generated data set containing the trigger into the trained protection model to ensure that only legal data can activate the model and carry out normal prediction. According to the method, the trigger is introduced into the frequency domain, the intellectual property is protected, meanwhile, high safety and concealment of the model are guaranteed, and the problems that in the prior art, a protection means is easily reversely extracted, the safety of the model is poor, and the attack resisting effect is poor can be effectively solved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of artificial intelligence security, and mainly relates to a frequency domain trigger driven deep learning model active protection method and system. BACKGROUND

[0002] With the wide application of deep neural networks in image recognition, natural language processing, automatic driving and other fields, the commercial value and social influence of deep learning models are increasing. However, the accompanying model intellectual property and privacy protection problems are also increasingly serious. The parameters and training data of deep learning models contain a large amount of business secrets and intellectual property rights, and if they cannot be effectively protected, the models will face the risk of illegal copying, reverse engineering and abuse. This not only may cause serious economic losses, but also may affect the competitiveness and market position of enterprises. The current model protection technology route can be divided into two kinds: passive protection and active protection.

[0003] Passive protection generally refers to marking the ownership of the model by embedding some unobtrusive marks or information. The literature Lv Haojie, Shen Shuyuan, Lin Huanjie, Yuan Yibo and Duan Delin. SVD Mark: A Novel Black-Box Watermarking for Protecting Intellectual Property of Deep Neural Network Model. International Conference on Artificial Intelligence and Security. Springer, Cham, 2022. uses the Mersenne-Twister algorithm to select watermark key samples, extracts the feature information of the key samples using the SVD algorithm, and constructs stable and hidden watermark samples to protect the security of deep models. The literature Yuan Zihan, Zhang Xinpeng, Wang Zichi and Yin Zhaoxia. Semi-Fragile Neural Network Watermarking Based on Adversarial Examples. in IEEE Transactions on Emerging Topics in Computational Intelligence, vol. 8, no. 4, pp. 2775-2790, Aug. 2024. The designed semi-fragile black-box watermarking scheme can effectively distinguish between accidental modification and malicious tampering of the model, thereby improving the usability assessment of the watermark model. Unlike passive protection, active protection not only embeds intellectual property information in the model, but also monitors, detects, and actively prevents illegal use or tampering of the model through technical means.Xue Mingfu, Wu Zhiyu, Zhang Yushu, Wang Jian and Liu Weiqiang. AdvParams: An active DNN intellectual property protection technique via adversarial perturbation based parameter encryption. in IEEE Transactions on Emerging Topics in Computing, vol. 11, no. 3, pp. 664-678, 1 July-Sept. 2023. Using an adversarial perturbation technique, encrypting part of the key parameters of the model can effectively prevent illegal use by malicious infringers.

[0004] Although the above-mentioned existing method provides a certain security, an attacker can still bypass the model verification mechanism by forging data that meets the license requirements, and ultimately achieve illegal use of the model, so we urgently need a method or system that can actively protect the model to improve the security and reliability of the model. SUMMARY

[0005] The present application is aimed at the problem that existing attackers can bypass the model verification mechanism by forging data that meets the license requirements, and achieve illegal use of the model. A frequency domain trigger driven deep learning model active protection method and system is proposed. First, white box access rights of the target model to be protected are obtained. The model dataset is obtained and the frequency domain trigger is generated. The generated frequency domain trigger is embedded in the dataset. The active protection model is trained to update the weight parameters in the model, so that the model relies on the trigger dataset. The generated dataset containing the trigger is input into the trained protection model to ensure that only legal data can activate the model and perform normal prediction. The present application introduces a trigger in the frequency domain to protect intellectual property rights while ensuring high security and concealment of the model. It can effectively solve the problems of easy reverse extraction of existing protection methods, poor model security, and poor adversarial attack effect.

[0006] To achieve the above-mentioned purpose, the technical solution adopted by the present application is: a frequency domain trigger driven deep learning model active protection method, comprising the following steps:

[0007] S1: Obtain white box access rights of a deep learning model, which are used to perform subsequent trigger embedding and dependency training. The deep learning model is a target model to be protected, and the white box access rights at least include structure and parameter information of the model;

[0008] S2: generating a frequency domain trigger according to the model data set, the frequency domain trigger being independent and unique for each data sample, the frequency domain trigger generation including an affine transformation, a nonlinear transformation and a frequency domain processing,

[0009] In the affine transformation, 3 pairs of data source points and data target points are selected from the data set to obtain an affine transformation image strongly related to the data sample, so as to establish a binding relationship between the trigger and the corresponding data sample;

[0010] In the nonlinear transformation, the anti-extraction and irreversibility of the trigger are enhanced through sine wave distortion and Gaussian noise;

[0011] In the frequency domain processing, the trigger is converted from the spatial domain to the frequency domain for subsequent trigger embedding;

[0012] S3: embedding the frequency domain trigger generated in step S2 into the data set, so that the high-frequency information of the frequency domain trigger is embedded into the frequency domain of the corresponding data sample;

[0013] S4: training the active protection model: after the frequency domain trigger generated in step S2 is embedded into the data set through step S3, a training sample set is formed, the model is actively protected and trained, the model parameters are iteratively optimized until the loss function converges, and an active protection model with trigger dependence is obtained;

[0014] S5: inputting the trigger data and non-trigger data obtained in step S3 into the active protection model obtained in step S4, wherein the input data carrying the legal trigger can activate the model and make a prediction, and the data without the trigger or with the wrong trigger cannot activate the model, thereby realizing the active protection of the model.

[0015] As an improvement of the present application, the target model to be protected in step S1 is any one of a deep neural network structure with trainable parameters, at least including a convolutional neural network, a residual network, a dense connection network and a wide residual network, and the pooling layer in the deep neural network structure with trainable parameters is a maximum pooling or uniform pooling, and the activation function form used in the activation layer includes but is not limited to ReLU, tanh.

[0016] As another improvement of the present application, in the affine transformation in step S2, the affine transformation function is specifically:

[0017] ;

[0018] wherein, represents an original image sample, represents a transformed image sample, represents an affine transformation operation;

[0019] The transformation matrix used in the affine transformation is specifically:

[0020] ;

[0021] wherein, is the transformed coordinate, denotes the original coordinate, denotes the affine transformation matrix, is a translation term.

[0022] As another improvement of the present application, in the nonlinear transformation in the step S2, the function of the nonlinear transformation is specifically:

[0023] ;

[0024] wherein, is a Gaussian noise with zero mean and variance ;

[0025] The sinusoidal wave distortion function is specifically:

[0026] ;

[0027] wherein, is the amplitude, is the frequency, is the phase.

[0028] As another improvement of the present application, in the frequency domain processing of the step S2, the frequency domain processing function is specifically:

[0029]

[0030]

[0031] wherein, and are the frequency domain representations of the original image and the affine transformed image, is the spatial domain pixel value of the image, and are the width and height of the image respectively, and are the frequency coordinates in the frequency domain, is the imaginary unit.

[0032] As another improvement of the present application, the embedding function of the frequency domain trigger generated in the step S3 is embedded into the data set, which is specifically:

[0033] ;

[0034] wherein, is the frequency domain representation of the original image, to represent high frequency information of the image for affine transformation, to be a weighting coefficient;

[0035] the embedded image is specifically:

[0036]

[0037] wherein, is the final image converted back to the spatial domain.

[0038] As a further improvement of the present application, the specific loss function of the model active protection training in step S4 is:

[0039]

[0040] wherein, represents the number of categories, is the real label of the th category, is the prediction probability of the model for the th category.

[0041] In order to achieve the above purpose, the technical scheme adopted by the present application is: a frequency domain trigger driven deep learning model active protection system, comprising a computer program, the computer program is executed by a processor to realize the steps of the method as claimed in any one of claims 1-7.

[0042] Compared with the prior art, the present application has the beneficial effects:

[0043] (1) The present application proposes a model active protection technology based on backdoor mechanism, which enables the model to recognize data carrying legal frequency domain triggers through training, and performs error classification processing on data not carrying or carrying incorrect triggers. Compared with existing methods, the present application has active identification and defense capability, and can prevent illegal data access in real time during the model running stage.

[0044] (2) The present application proposes a data-specific frequency domain trigger generation mechanism, which makes each data sample correspond to a unique frequency domain trigger, and combines sample features for customized design, thereby realizing high correlation between the trigger and the data. This design enhances the anti-reverse ability of the protection system, effectively prevents attackers from extracting and reusing trigger information through reverse engineering, visualization or generalization technology, and ensures the security and intellectual property integrity of the model. At the same time, this mechanism improves the uniqueness and unforgeability of the trigger, further improving the security protection level of the deep model in practical application.

[0045] (3) The method has good universality and deployment compatibility without changing the original model structure, can control the accuracy loss within an acceptable range under the premise of ensuring model security, and has high practical value. BRIEF DESCRIPTION OF DRAWINGS

[0046] Figure 1 is a structural schematic diagram of a target model to be protected by the method of the present application;

[0047] Figure 2 is a comparison schematic diagram of images with and without triggers in the method of the present application;

[0048] Figure 3 is a combination form schematic diagram of training data samples in step S4 in embodiment 1 of the present application;

[0049] Figure 4 is a T-SNE visualization classification diagram of the protected model of the present application;

[0050] Figure 5 is a schematic diagram of expected results of the protected model in the test example of the present application. DETAILED DESCRIPTION

[0051] The present application will be further illustrated in conjunction with the drawings and specific embodiments, and it should be understood that the following specific embodiments are only used to illustrate the present application and not to limit the scope of the present application.

[0052] Embodiment 1

[0053] A frequency domain trigger driven deep learning model active protection method, comprising the following steps:

[0054] Step S1, obtaining white box access permission of the deep learning model: including model structure and parameter information, used for performing trigger embedding and dependent training and other protection design operations;

[0055] The target model to be protected is not limited to a specific network structure form, and can be a convolutional neural network, a residual network, a dense connection network, a wide residual network or other deep neural network structure with trainable parameters; the pooling layer is any one of maximum pooling and uniform pooling; the activation function form used by the activation layer includes but is not limited to ReLU, tanh.

[0056] Step S2, obtaining the model data set and generating the frequency domain trigger: the frequency domain trigger is independently designed and unique for each data sample to ensure the specificity of the trigger and the effectiveness of the protection, and the frequency domain trigger generation includes affine transformation, nonlinear transformation and frequency domain processing,

[0057] In the affine transformation, 3 pairs of data source points and data target points are selected from the data set, and then an affine transformation image strongly related to the data sample is obtained. The trigger is constructed by using the affine transformation, so that the trigger is closely bound to the corresponding data sample, ensuring the uniqueness and irreplaceability of the trigger. In the nonlinear transformation, the sine wave distortion and Gaussian noise are introduced to enhance the anti-extraction and irreversibility of the trigger. In the frequency domain processing, the trigger can be converted from the spatial domain to the frequency domain for subsequent embedding operation.

[0058] The structural diagram of the protection target model is shown in Figure 1 The overall framework mainly includes a trigger generation module, a trigger embedding module, and a trigger training module. The trigger generation module is used to generate a frequency domain trigger strongly related to a data sample. The trigger embedding module is used to embed the generated frequency domain trigger into the frequency domain of the original image data. This module extracts the high-frequency components in the trigger and embeds them into the corresponding frequency domain position of the image. On the premise of ensuring the integrity of the image content, the embedding strength of the trigger is adaptively controlled, so that the embedded image is visually consistent with the original image while having strong anti-attack ability. The trigger training module is used to introduce a trigger-dependent mechanism during model training, so that the target model can only be normally activated and perform inference tasks when the input data carries the correct trigger. When the input data does not carry the trigger or carries the wrong trigger, the target model cannot work normally, thereby realizing access control of the model function and effectively preventing the model from being used or misused without authorization.

[0059] The function of the affine transformation in step S2 is:

[0060]

[0061] wherein, represents the original image sample, represents the transformed image sample, represents the affine transformation operation.

[0062] The transformation matrix used in the affine transformation in step S2 is:

[0063]

[0064] wherein, is the transformed coordinate, represents the original coordinate, represents the affine transformation matrix, is a translation term. In this way, each image will produce a unique affine transformation.

[0065] The function of the nonlinear transformation introduced in step S2 is:

[0066]

[0067] wherein, is a zero-mean and variance Gaussian noise, and the sinusoidal wave distortion function introduced later is:

[0068]

[0069] wherein, is the amplitude, is the frequency, is the phase, through these nonlinear transformations, the trigger improves the resistance.

[0070] The frequency domain processing function in step S2 is:

[0071]

[0072]

[0073] wherein, and are the frequency domain representations of the original image and the affine transformed image, is the spatial domain pixel value of the image, and are the width and height of the image respectively, and are the frequency coordinates in the frequency domain, is the imaginary unit.

[0074] Step S3, using the generated frequency domain trigger to embed in the data set: the high frequency information of the frequency domain trigger is embedded into the frequency domain of the corresponding data sample; the processing ensures that the trigger is hidden while the overall content of the original image is minimally affected.

[0075] The embedding function of the generated frequency domain trigger embedded in the data set is:

[0076]

[0077] wherein, is the frequency domain representation of the original image, is the high frequency information representation of the affine transformed image, is the weighting coefficient, which controls the degree of influence of the high frequency information on the original image.

[0078] The final image after the frequency domain trigger is embedded in step S3 is specifically:

[0079]

[0080] wherein, The final image is converted back to the spatial domain.

[0081] Figure 2 This is a schematic diagram comparing images of data samples without triggers and data samples with triggers, visually illustrating the impact of the frequency domain trigger embedding method on the visual characteristics of the original data in this invention. For example... Figure 2 As shown, the data samples with embedded frequency domain triggers maintain a high degree of visual consistency with the original clean data, making them difficult to distinguish with the naked eye. This indicates that the frequency domain triggers have good concealment in the spatial domain. This comparative result demonstrates that the present invention, by embedding triggers into the high-frequency components of the data in the frequency domain, achieves effective trigger embedding without significantly altering the original data content and visual characteristics, thus balancing model protection and data usability.

[0082] Step S4: Train the active protection model: After embedding the frequency domain trigger generated in step S2 into the dataset in step S3, a training sample set is formed to train the model for active protection. The model parameters are iteratively optimized until the loss function converges, and an active protection model with trigger dependency is obtained.

[0083] The specific loss function for model training in step S4 is as follows:

[0084]

[0085] in, Indicates the number of categories. For the first Real labels for each category For the model number The predicted probability of a class.

[0086] In this embodiment, the training data sample combination is as follows: Figure 3 The diagram illustrates the trigger-dependent training strategy employed by this invention during the model training phase. Figure 3As can be seen, three different data sets are constructed during the training process to jointly constrain the learning behavior of the target model. Among them, data set A contains original data samples embedded with frequency domain triggers, and the corresponding labels remain correct, used to guide the model to establish normal prediction ability for legal trigger data. Data set B is composed of original data samples without embedded frequency domain triggers, and the corresponding labels are randomly shuffled, used to simulate unauthorized data input scenarios. By imposing false label constraints on this type of data, the model is prevented from learning effective prediction mapping relationships in the absence of triggers, thereby weakening the model's response to unauthorized data. In data set C, the original data samples are embedded with randomly shuffled frequency domain triggers, so that each sample carries a trigger that is inconsistent with its proper trigger, and its label is also randomly processed. This data set is used to further constrain the model's response to false trigger inputs and prevent the model from being activated by false triggers. Through the collaborative training of the above three types of data sets, the target model is forced to form a strong dependence on the correct frequency domain trigger, and only in the case of input data carrying legal triggers can the model obtain correct prediction results, thereby achieving effective access control and protection of the model's function.

[0087] Step S5, result output: input the trigger data and non-trigger data obtained in step S3 into the trained active protection model, so that input data carrying legal triggers can activate the model and perform normal prediction, while data without triggers or with false triggers cannot activate the model.

[0088] Figure 4 The visualization results of the ResNet-18 model processed by T-SNE on the CIFAR-10 data set without trigger data and with correct trigger data are analyzed. When the input data carries the correct frequency domain permission trigger, the model can form a clear and separable class clustering structure in the feature space, indicating that the model maintains good classification discrimination ability under the condition of permission data and can accurately distinguish different categories of data samples. In contrast, when the input data is unauthorized clean data without embedded triggers, the corresponding feature distribution of the model presents a highly chaotic and random state, and the samples of each category are difficult to form effective clustering in the feature space, indicating that the model cannot effectively classify unauthorized data.

[0089] Test Example

[0090] To verify the universality and effectiveness of the scheme under different model structures and different data set conditions, the comparative experiment is carried out in the unified experimental environment. The experiment is implemented based on the PyTorch deep learning framework and trained and tested on multiple NVIDIA Tesla V100 GPUs. The MNIST, CIFAR-10 and CIFAR-100 are selected as the benchmark data sets. Among them, the MNIST data set contains 70,000 28x28 pixel handwritten digital gray scale images; the CIFAR-10 data set contains 60,000 32x32 pixel color images, covering 10 categories; the CIFAR-100 data set also contains 60,000 images, but the number of categories is expanded to 100. In terms of model selection, the test example adopts typical deep neural network structures such as VGG16, ResNet18, DenseNet121 and WideResNet50-2 to verify the applicability of the scheme under different model architectures. All models are trained using the cross-entropy loss function, and the specific training parameter settings are shown in Table 1:

[0091] Table 1 Training parameter settings

[0092]

[0093] To construct the comparative example, under the same data set, the same model structure and the same training parameter conditions, the following two training and testing settings are carried out respectively:

[0094] Comparative example: the model only uses the original clean data for regular training and testing, without introducing any frequency domain trigger and trigger dependent training mechanism;

[0095] Embodiment (method of the application): the model introduces the frequency domain trigger generation and embedding according to the method of the application, and combines the trigger dependent training strategy for training, and evaluates the data carrying the correct trigger and the unauthorized data in the test stage respectively.

[0096] In the present test example, first, the frequency domain trigger strongly related to the data sample is generated according to the method of the present application and embedded into the training data; then, the training data combination containing authorized data and unauthorized data is constructed, and the trigger-dependent training is performed on the target model. In the test stage, the classification accuracy of the model under the following two input conditions is counted respectively: the classification accuracy when the input data carries the correct frequency domain trigger / authorized data; the classification accuracy when the input data does not carry the trigger or carries the wrong trigger / unauthorized data. The prediction accuracy of the model is shown in Table 2. The accuracy results shown in Table 2 are obtained by averaging the results of multiple experiments on the corresponding model and data set combination under the above two test conditions. Acc-Clean is the initial prediction accuracy of the model trained on the clean data set, Acc-Pro is the prediction accuracy of the protected model on the data set containing the frequency domain trigger, and Acc-Origin is the prediction accuracy of the protected model on the clean data set.

[0097] Table 2

[0098]

[0099] Figure 5 For comparison of the classification accuracy of the protected model and the unprotected model on the authorized data set and the unauthorized data set, Table 1 is a numerical summary of the corresponding experimental results. In combination with Figure 5 As can be seen from Table 1, when the present application scheme is used, each type of model can maintain a high classification accuracy on data carrying the correct frequency domain trigger, which is basically consistent with the performance of the benchmark model in the comparative example, indicating that the introduction of the frequency domain trigger does not significantly affect the normal classification ability of the model. At the same time, when the model processes unauthorized data, its classification accuracy decreases significantly and in most cases is close to the level of random guessing, which is significantly lower than the performance of the model in the comparative example under the same data conditions. This result shows that through the frequency domain trigger and trigger-dependent training mechanism proposed by the present application, the target model forms a significant dependence relationship on authorized data at the functional level.

[0100] In combination with the above method process, the core idea of the frequency domain trigger driven active protection method involved in the present application is to generate a unique frequency domain trigger for each training data sample and embed it in the frequency domain of the data, thereby establishing a strong binding relationship between the data and the trigger. During the model training process, the guide model only responds to input data carrying the correct trigger, so that the model cannot be normally activated when facing data that does not carry or carries the wrong trigger. Through this mechanism, the present application realizes the control of model access rights through input data under the premise of unchanged model structure, achieving the purpose of active protection of deep learning models. Overall, the present application provides a knowledge property protection scheme with high concealment, high resistance and universality, which can effectively prevent unauthorized data access and model abuse, and solves the problems of weak protection effect, difficulty in preventing reverse extraction and inability to actively control model access of traditional watermarking technology.

[0101] It should be noted that the above content only illustrates the technical idea of the present application and cannot limit the protection scope of the present application. For ordinary skilled persons in the technical field, they can make several improvements and refinements without departing from the principles of the present application, and these improvements and refinements fall within the protection scope of the claims of the present application.

Claims

1. A method for proactive protection of a deep learning model driven by a frequency domain trigger, characterized in that: The method comprises the following steps: S1: obtaining white box access permission of a deep learning model for performing subsequent trigger embedding and dependence training; the deep learning model is a target model to be protected, and the white box access permission at least comprises structure and parameter information of the model; S2: generating a frequency domain trigger according to a model data set, the frequency domain trigger being independent and unique for each data sample, the frequency domain trigger generation comprising affine transformation, nonlinear transformation and frequency domain processing, in the affine transformation, 3 pairs of data source points and data target points are selected from the data set to obtain an affine transformation image strongly related to the data sample, so as to establish a binding relationship between the trigger and the corresponding data sample; in the nonlinear transformation, the trigger is enhanced in anti-extraction and irreversibility through sine wave distortion and Gaussian noise; in the frequency domain processing, the trigger is converted from the spatial domain to the frequency domain for subsequent trigger embedding; S3: embedding the frequency domain trigger generated in step S2 into the data set, so that high-frequency information of the frequency domain trigger is embedded into the frequency domain of the corresponding data sample; S4: training an active protection model: after the frequency domain trigger generated in step S2 is embedded into the data set through step S3, a training sample set is formed, the model is actively protected and trained, the model parameters are iteratively optimized, and the loss function is converged until the loss function is converged, and an active protection model with trigger dependence is obtained; S5: inputting the trigger data obtained in step S3 and the non-trigger data into the active protection model obtained in step S4, wherein the input data carrying the legal trigger can activate the model and perform prediction, and the data without the trigger or with the wrong trigger cannot activate the model, thereby realizing active protection of the model. 2.The method of claim 1, wherein: The target model to be protected in step S1 is any one of a deep neural network structure with trainable parameters, at least comprising a convolutional neural network, a residual network, a dense connection network and a wide residual network, and the pooling layer in the deep neural network structure with trainable parameters is a maximum pooling or uniform pooling, and the activation function form used in the activation layer includes but is not limited to ReLU and tanh. 3.The method of claim 1, wherein: In the affine transformation in step S2, the affine transformation function is specifically: ； wherein, denotes the original image sample, denotes the transformed image sample, denotes an affine transformation operation; The transformation matrix used in the affine transformation is specifically: ； wherein, is the transformed coordinate, denotes the original coordinate, denotes the affine transformation matrix, is the translation term. 4.The method of claim 3, wherein the method further comprises: determining whether the frequency domain trigger is activated; and if the frequency domain trigger is activated, performing the deep learning model active protection. In the nonlinear transformation in step S2, the function of the nonlinear transformation is specifically: ； wherein is a zero-mean Gaussian noise with variance is a zero-mean Gaussian noise with variance The sine wave distortion function is specifically: ； wherein, is the amplitude, is the frequency, is the phase. 5.The method of claim 3, wherein: In the frequency domain processing of step S2, the frequency domain processing function is specifically: ； ； wherein and are the frequency domain representations of the original image and the affine transformed image, is the spatial domain pixel value of the image, and are the width and height of the image, respectively, and are the frequency coordinates in the frequency domain, is the imaginary unit. 6.The method of claim 5, wherein the method further comprises: determining whether the frequency domain trigger is activated; and if the frequency domain trigger is activated, performing the deep learning model active protection. The embedding function of the frequency domain trigger generated in step S3 into the data set is: ； wherein is a frequency domain representation of the original image, is a high frequency information representation of the affine transformed image, is a weighting factor; The embedded image is specifically: ； wherein is the final image for conversion back to the spatial domain. 7.The method of claim 1, wherein: The specific loss function of the model active protection training in step S4 is: ； wherein, represents the number of classes, is the true label of the th class, is the predicted probability of the model for the th class. 8.A deep learning model active protection system driven by frequency domain trigger, comprising a computer program, characterized in that: The computer program is executed by the processor to realize the steps of the method in any one of claims 1-7.

Images