An adaptive neural network watermarking method with intra-parameter and inter-parameter mutual verification

Abstract

The application discloses a self-adaptive neural network watermarking method for mutual verification within parameters and between parameters, acquires a model file to be protected, and traverses each layer of the model according to the model file, and uses a secret key to disarrange the order of each layer parameter; then, inter-parameter bit exchange adds inter-parameter reversible watermark through reversible calculation, then generates adaptive bits for restoring the performance of the model and generates self-checking intra-parameter watermark, and finally completes watermark adding. The application uses the maximum increase of watermark capacity through the adaptive method, while ensuring the model accuracy. Through the calculation between parameters and within parameters, the calculation of the watermark itself can detect the modification of any parameter and realize accurate recovery.

Description

Technical Field

[0001] This invention relates to computer artificial intelligence model integrity protection technology, specifically to an adaptive neural network watermarking method that verifies both intra-parameter and inter-parameter parameters. Background Technology

[0002] Currently, artificial intelligence (AI) is developing rapidly, and AI models have penetrated into all aspects of life, with ChatGPT being one of the most popular examples recently. Consequently, the protection of AI models has become a hot topic. The protection of AI models can be broadly divided into two aspects: 1) Protecting the model's copyright by adding specific tags to the model files to prove ownership. 2) Protecting the model's integrity by adding protective modules (called fragile watermarks) to the model information to detect whether the model has been tampered with. Tampering often leads to abnormal model functions, resulting in greater security risks after deployment.

[0003] Currently, the relevant existing technologies are mainly based on neural network white-box fragile watermarking. These methods are often simply migrated from the field of multimedia watermarking without properly combining the characteristics of neural networks, which results in the following defects: 1) The embedded watermark capacity is insufficient, and embedding often leads to a decrease in performance; 2) The detection accuracy of existing watermarking technologies is insufficient. Most of them protect the parameters after they are divided into blocks, and the accuracy can only reach the block level, and the recovery performance is often poor.

[0004] For example, although the paper "DNN self-embedding watermarking: Towards tampering detection and parameter recovery for deep neural network" (Zhao G et al.) discloses a similar method for the security protection and integrity verification of neural network models, this existing solution cannot be adaptive, which leads to a significant performance loss when encountering deep networks. At the same time, the localization capability of this technical solution is at the block level.

[0005] For example, patent CN 111861849 A discloses a method for embedding watermark information into an artificial intelligence model, but the purpose of embedding the watermark in this technical solution is to protect the copyright of the model and to make the watermark as indiscriminate as possible.

[0006] For example, patent CN_115408699_A discloses a method and system for security protection and integrity verification of neural network models. Although it can identify the integrity of the model, it cannot recover the tampered model. Summary of the Invention

[0007] Purpose of the invention: The purpose of this invention is to overcome the shortcomings of the existing technology and provide an adaptive neural network watermarking method that verifies both intra-parameter and inter-parameter parameters.

[0008] Technical solution: The present invention provides an adaptive neural network watermarking method for mutual verification within and between parameters, comprising the following steps:

[0009] Step S1: Obtain the model file to be protected, and traverse each layer of the model according to the model file, shuffling the order of the parameters of each layer using a key; each parameter consists of a 32-bit floating-point number, denoted as b0, b1, b2, ... b 31 The model file includes the model's structural and parameter information.

[0010] This avoids the need to scramble the parameters using a key at the beginning to prevent the watermark information from being copied. On the one hand, it ensures that if the outside world wants to reproduce the watermark, it cannot reproduce it without the key. On the other hand, it ensures that when the watermark is used to restore the model performance later, the external disturbance is applied as evenly as possible to the scrambled parameter sequence.

[0011] Step S2: For the 32-bit floating-point parameters of the current processing layer after being shuffled in step S1, perform bit swapping between parameters. This allows for the addition of a reversible watermark between parameters to the current processing layer through reversible calculation, achieving parameter backup. The specific method is as follows:

[0012] The shuffled parameter sequence is renumbered, with the first digit considered 0 and the last digit considered N-1 (the parameter preceding the 0th parameter is N-1, and similarly, the parameter following the 1st parameter is the 0th parameter). The preceding and following parameters of each parameter are then identified. The calculation begins with the parameter numbered 0, calculating its b0, b1, b2, ... b... 10 XORed with an 11-bit key and then ANDed with parameters numbered 1, b0, b1, b2, ... b 10 Replace bit number 1 with the obtained 11 bits. 12 ,b 13 ,b 14 ,…b 22 A total of 11 bits are used. Calculate the 11 bits generated by number 1, the key, and number 2, and replace the 'b' in parameter number 2. 12 ,b 13 ,b 14 ,…b 22 bits;

[0013] The next layer of watermarks will be added only after all parameters of the current processing layer, whose watermarks are determined by the relationships between the parameters, have been added.

[0014] Step S3: Generate adaptive bits for restoring model performance.

[0015] First, replace 11 bits of each parameter: b 12 ,b 13 ,b 14 ,…b 22 ; its b0, b1, b2, ... b 10 It is the original bit, b 11 The bit is an adaptive bit that is not replaced, and then it is modified according to the principle of gradient descent; the modification rules include the following four cases:

[0016] a) When the tensor is "-" and the gradient is "-", adaptive b 11 The bit is 0;

[0017] b) When the tensor is "-" and the gradient is "+", adaptive b 11 The bit is 1;

[0018] c) When the tensor is "+" and the gradient is "-", adaptive b 11 The bit is 1;

[0019] d) When the tensor is "+" and the gradient is "+", adaptive b 11 The bit is 0;

[0020] Step S4: Generate a self-testing parameter watermark.

[0021] Based on the values ​​of b0, b1, b2, ... b obtained from steps S1 to S3 22 Bits, the final b is calculated using a hash function. 23 ,b 24 ,b 25 ,…b 31 The nine-bit watermark is added by performing a modulo operation on the first 23 bits of the binary data and then replacing the last nine bits with the resulting value.

[0022] Furthermore, it also includes the detection of watermarks within and between parameters, and the verification method is as follows:

[0023] First, scramble the keys with the same identifier, then perform the same XOR operation as when adding the watermark to verify that the calculated result matches b0, b1, b2, ... b 10 If they are the same, then it is directly determined that the number has been tampered with. If they are the same, then the modulo operation of the first 23 bits is performed to confirm whether the result and the value recorded in the last 9 bits represent the same number.

[0024] Furthermore, if the detection indicates that the data has been tampered with, the model file is restored. The restoration method is as follows: when an error is encountered in the watermark between parameters, the modulo hash operation is checked to determine which of the two parameters used to generate the watermark between parameters is correct and which is incorrect. If the last 9 bits of both parameters are different when calculating the hash function, it is determined that the data cannot be restored. If only one parameter is found to be incorrect, the other parameter is used for restoration.

[0025] Furthermore, the gradient in step S3 is obtained from the training set through backpropagation, and the sign of the tensor is determined by the already trained model itself. The specific method for modifying the adaptive bit according to the gradient descent principle is as follows:

[0026] First, select two hyperparameters α and β, which represent the number of training rounds required for each layer and the proportion of parameter changes required for each layer, respectively (note that these two parameters need to be set in vector form when dealing with large models to ensure that the number of training rounds and the proportion of parameter changes for each layer are appropriate).

[0027] Then, each layer is traversed; during the α-th training, the gradients of all parameters of the current layer are obtained through the training set, and the accuracy of the model at this time is obtained through the test set;

[0028] Next, the corresponding parameters are sorted in descending order of gradient. The first β% of the sorted parameters are then calculated, and it is determined whether (tensor × gradient) is greater than 0. If it is greater than 0, then the b-value of that parameter is... 11 If the bit is equal to 0, or not greater than 0, then the parameter b... 11 The bit is equal to 1; finally, after each layer is processed, the accuracy information at this time is recorded using the test set for comparison with the state before adjustment. If it is better, the state of the current processing layer is switched to the adjusted parameter state. This process is repeated in a loop, and finally the most ideal adaptive parameters are saved.

[0029] In step S3 above, adaptive b 11The purpose of the bit is to simulate gradient descent and select the correct step size and direction. For example, for parameters with positive gradients, we want to decrease their size to reduce the loss function value; for parameters with negative gradients, we want to increase their size. To reduce the adaptive adjustment step size, the gradients are arranged from largest to smallest. β is selected as the percentage of parameters to be adjusted, and α is selected as the number of training epochs for each layer. The best-performing layer during training is selected and saved to obtain the final adaptive bit. In other words, after selecting each layer, other layers are frozen. Gradient information is obtained using the training set, while the original accuracy information is obtained using the test set to prevent overfitting and ensure the model's generalization ability. After obtaining the gradients, the parameters are arranged. The top β percent of parameters undergo adaptive processing as shown in Table 1. After each layer is processed, the accuracy information at this point is recorded using the test set for comparison with the state before adjustment. If the result is better, the state of the current layer is switched to the adjusted parameter state. This process is repeated until the most ideal adaptive parameters are saved.

[0030] Beneficial Effects: This invention utilizes an adaptive method to significantly increase watermark capacity while maintaining model accuracy. For the watermark calculation itself, through inter-parameter and intra-parameter operations, it ensures that modifications to any parameter can be detected and accurate recovery can be achieved. The embedded watermark capacity of this invention can reach 62.5% of the total number of bits in the model parameters. Each parameter has 32 bits, with a maximum of 20 bits embedded. That is, if all parameters are protected, 62.5% of the total bits occupied by all parameters can be used as a watermark without performance degradation. Simultaneously, this invention achieves parameter-level accuracy in detecting and locating tampering positions, enabling 100% detection. Attached Figure Description

[0031] Figure 1 This is a schematic diagram of the overall process of generating a watermark in this invention;

[0032] Figure 2 This is a schematic diagram of the neural network structure parameters of the present invention;

[0033] Figure 3 This is a schematic diagram illustrating the generation of adaptive bits in this invention;

[0034] Figure 4 This is a schematic diagram of the detection and recovery process in this invention;

[0035] Figure 5 This is a schematic diagram illustrating the comparison of model performance recovery across four datasets when facing an attack, as shown in the example. Detailed Implementation

[0036] The technical solution of the present invention will be described in detail below, but the scope of protection of the present invention is not limited to the embodiments described.

[0037] like Figure 1 As shown, the adaptive neural network watermarking method of the present invention, which verifies both intra-parameter and inter-parameter parameters, includes the following steps:

[0038] Step S1: Obtain the model file to be protected, and traverse each layer of the model according to the model file, shuffling the order of the parameters of each layer using a key; the model file includes the model's structural information and parameter information; each parameter consists of a 32-bit floating-point number, denoted as b0, b1, b2, ... b 31 ; Figure 2 This embodiment shows the structure of the parameters after the watermark is added, including the first information bit, the middle adaptive bit, the watermark bit between parameters, and the hash watermark bit inside the last parameter.

[0039] Step S2: For the 32-bit floating-point parameters of the current processing layer after being shuffled in step S1, perform bit swapping between parameters to add a reversible watermark between parameters to the current processing layer through reversible calculation. The specific method is as follows:

[0040] First, renumber the shuffled parameter sequence, starting with 0 and ending with N-1, and find the preceding and following parameters for each parameter. Then, start calculating the parameter numbered 0, setting its b0, b1, b2, ... b... 10 XORed with an 11-bit key and then ANDed with parameters numbered 1, b0, b1, b2, ... b 10 Replace bit number 1 with the obtained 11 bits. 12 ,b 13 ,b 14 ,…b 22 A total of 11 bits;

[0041] Next, calculate the 11 bits generated by number 1, the key, and number 2, and replace the b corresponding to parameter number 2. 12 ,b 13 ,b 14 ,…b 22 The bits; until all parameters of the current processing layer have been added with watermarks determined by the relationships between the parameters, the next layer of watermarks will be added.

[0042] Step S3: Generate adaptive bits for restoring model performance.

[0043] First, replace 11 bits of each parameter: b 12 ,b 13 ,b 14 ,…b 22 ; its b0, b1, b2, ... b 10 It is the original bit, b 11The bit is an adaptive bit that is not replaced, and then it is modified according to the principle of gradient descent; the modification rules include the following four cases:

[0044] a) When the tensor is "-" and the gradient is "-", adaptive b 11 The bit is 0;

[0045] b) When the tensor is "-" and the gradient is "+", adaptive b 11 The bit is 1;

[0046] c) When the tensor is "+" and the gradient is "-", adaptive b 11 The bit is 1;

[0047] d) When the tensor is "+" and the gradient is "+", adaptive b 11 The bit is 0;

[0048] The specific details are shown in Table 1;

[0049] Table 1 Modification Rules

[0050] - - 0 - + 1 + - 1 + + 0

[0051] Step S4: Generate a self-testing parameter watermark.

[0052] Based on the values ​​of b0, b1, b2, ... b obtained from steps S1 to S3 22 The final bit b is obtained by calculating the hash function from the bits. 23 ,b 24 ,b 25 ,…b 31 The nine-bit watermark is added by performing a modulo operation on the first 23 bits of the binary data and then replacing the last nine bits with the resulting value.

[0053] like Figure 3 As shown, in this embodiment, the gradient is obtained from the training set through backpropagation. The sign of the tensor is determined by the already trained model itself. The specific method for modifying the adaptive bit according to the gradient descent principle is as follows:

[0054] First, select two hyperparameters α and β, which represent the number of training rounds required for each layer and the proportion of parameters that need to be changed for each layer, respectively.

[0055] Then, each layer is traversed; during the α-th training, the gradients of all parameters of the current layer are obtained through the training set, and the accuracy of the model at this time is obtained through the test set;

[0056] Next, the corresponding parameters are sorted in descending order of gradient. The first β% of the sorted parameters are then calculated, and it is determined whether (tensor × gradient) is greater than 0. If it is greater than 0, then the b-value of that parameter is... 11 If the bit is equal to 0, or not greater than 0, then the parameter b... 11 The bit is equal to 1; finally, after each layer is processed, the accuracy information at this time is recorded using the test set for comparison with the state before adjustment. If it is better, the state of the current processing layer is switched to the adjusted parameter state. This process is repeated in a loop, and finally the most ideal adaptive parameters are saved.

[0057] This embodiment detects watermarks within and between parameters.

[0058] First, scramble the keys with the same identifier, then perform the same XOR operation as when adding the watermark to verify that the calculated result matches b0, b1, b2, ... b 10 If the values ​​are the same, the file is considered tampered with. If they are different, it is directly determined that the file has been tampered with. If they are the same, a modulo operation is performed on the first 23 bits to confirm whether the result is the same as the value recorded in the last 9 bits. If the detection indicates tampering, the model file is restored. The restoration method is as follows: when an error is encountered in the watermark between parameters, the modulo hash operation is checked to determine which of the two parameters used to generate the watermark is correct and which is incorrect. If both parameters are found to be different in the last 9 bits when calculating the hash function, it is determined that the file cannot be restored. If only one parameter is found to be incorrect, restoration is performed using the other parameter.

[0059] Specific detection and recovery methods are as follows: Figure 4 As shown:

[0060] 1) Traverse each layer of the neural network;

[0061] 2) Restore the order of the encryption keys used in the current layer, and define the parameter labeled 0. The preceding parameter is the last parameter in this parameter sequence, i.e., the parameter following the last parameter, and is defined as the parameter with index 0. Begin using reversible computation, i.e., the XOR operation, to... Figure 2 The Nth and (N+1)th information bits in the key are XORed with each other, and the result is then ANDed with the key. Figure 2 The cross-check bits of the N+1 bits of the parameter are compared to check if they are the same;

[0062] 3) If they are the same, then perform a self-checking bit verification for each parameter. Specifically, the same as encryption, the binary value formed by the first 23 bits is hashed using the same method as encryption, in order to further check whether it has been tampered with.

[0063] 4) If any of the above checks fails, it will be determined that the device has been tampered with;

[0064] 5) If you want to restore the model, you need to perform a self-check on the two parameters involved in the mutual check after the mutual check parameter check fails. If both parameters fail, then the parameter with the error is pruned and set to zero. If there is only one error, then the information of the correct parameter can be restored to the corresponding information bit of the parameter with the error through the reversible operation above.

[0065] Example

[0066] This embodiment was tested on four models from shallow to deep. The tests proved that the adaptability of the technical solution of the present invention can well recover the model and the performance degradation caused by adding watermarks. In addition, the recovery performance of the method of the present invention can still recover the model performance well when the attacked parameters are around 20%.

[0067] Table 2. Performance comparison of the present invention with self-embedding watermarking

[0068]

[0069] Table 2 shows the performance metrics for existing self-embedding watermarking methods in the PRL2022 column, and the performance metrics for the technical solution of this invention in the Ours column. The data in Table 2 shows that for models with a larger number of layers, the performance of existing self-embedding watermarking methods degrades more, while this invention consistently maintains good model performance. Comparing the accuracy rates of clean models (without watermarking), before adaptive models (with watermarking but not yet adaptive watermarking), and after adaptive models (with adaptive watermarking), it can be seen that this invention has a significant recovery capability for deep models.

[0070] like Figure 5 As shown, the technical solution of this invention recovers model performance across four datasets in the face of attacks with different attack ratios (tampered rated). Among them, Figure 5 a is a schematic diagram of the LeNet model. Figure 5 b is a schematic diagram of the AlexNet model. Figure 5 c is a schematic diagram of the ResNet18 model. Figure 5 d is a schematic diagram of the ResNet50 model.

[0071] As can be seen from the conclusions of the above embodiments, the present invention utilizes an adaptive method to significantly increase the watermark capacity while maintaining model accuracy; furthermore, the present invention can detect external changes to the watermark and thus determine if the model has been altered, identifying whether the model remains intact. Simultaneously, the present invention achieves performance not degraded after embedding the watermark through adaptive technology.

Claims

1. An adaptive neural network watermarking method for mutual verification within and between parameters, characterized in that, Includes the following steps: Step S1: Obtain the model file to be protected, and traverse each layer of the model according to the model file, shuffling the order of the parameters of each layer using a key; the model file includes the model's structural information and parameter information; each parameter consists of a 32-bit floating-point number, denoted as b0, b1, b2, ... b 31 ; Step S2: For the 32-bit floating-point parameters of the current processing layer after being shuffled in step S1, perform bit swapping between parameters to add a reversible watermark between parameters to the current processing layer through reversible calculation. The specific method is as follows: First, renumber the shuffled parameter sequence, starting with 0 and ending with N-1, and find the preceding and following parameters for each parameter. Then, start calculating the parameter numbered 0, setting its b0, b1, b2, ... b... 10 XORed with an 11-bit key and then ANDed with parameters numbered 1, b0, b1, b2, ... b 10 Replace bit number 1 with the obtained 11 bits. 12 ,b 13 ,b 14 ,…b 22 A total of 11 bits; Next, calculate the 11 bits generated by number 1, the key, and number 2, and replace the b corresponding to parameter number 2. 12 ,b 13 ,b 14 ,…b 22 The bits; until all parameters of the current processing layer have been added with watermarks determined by the relationships between the parameters, the next layer of watermarks will be added. Step S3: Generate adaptive bits for restoring model performance. First, replace 11 bits of each parameter: b 12 ,b 13 ,b 14 ,…b 22 ; its b0, b1, b2, ... b 10 It is the original bit, b 11 The bit is an adaptive bit that is not replaced, and then it is modified according to the principle of gradient descent; the modification rules include the following four cases: a) When the tensor is "-" and the gradient is "-", adaptive b 11 The bit is 0; b) When the tensor is "-" and the gradient is "+", adaptive b 11 The bit is 1; c) When the tensor is "+" and the gradient is "-", adaptive b 11 The bit is 1; d) When the tensor is "+" and the gradient is "+", adaptive b 11 The bit is 0; Step S4: Generate a self-testing parameter watermark. Based on the values ​​of b0, b1, b2, ... b obtained from steps S1 to S3 22 The final bit b is obtained by calculating the hash function from the bits. 23 ,b 24 ,b 25 ,…b 31 The nine-bit watermark is added by performing a modulo operation on the first 23 bits of the binary data and then replacing the last nine bits with the resulting value.

2. The adaptive neural network watermarking method for mutual verification within and between parameters according to claim 1, characterized in that, It also includes the detection of watermarks within and between parameters, and the verification method is as follows: First, scramble the keys with the same identifier, then perform the same XOR operation as when adding the watermark to verify that the calculated result matches b0, b1, b2, ... b 10 If they are the same, then it is directly determined that the number has been tampered with. If they are the same, then the modulo operation of the first 23 bits is performed to confirm whether the result and the value recorded in the last 9 bits represent the same number.

3. The adaptive neural network watermarking method for mutual verification within and between parameters according to claim 1 or 2, characterized in that, If the detection indicates that the model file has been tampered with, the recovery process will be performed. The recovery method is as follows: When an error occurs in the watermark between parameters, the modulo hash operation is checked to determine which of the two parameters used to generate the watermark is correct and which is incorrect. If the last 9 bits of both parameters are different when the hash function is calculated, it is determined that the watermark cannot be recovered. If only one parameter is found to be incorrect, the other parameter is used to recover the watermark.

4. The adaptive neural network watermarking method for mutual verification within and between parameters according to claim 1, characterized in that, In step S3, the gradient is obtained from the training set through backpropagation. The sign of the tensor is determined by the already trained model itself. The specific method for modifying the adaptive bit according to the gradient descent principle is as follows: First, select two hyperparameters α and β, which represent the number of training rounds required for each layer and the proportion of parameters that need to be changed for each layer, respectively. Then, each layer is traversed; during the α-th training, the gradients of all parameters of the current layer are obtained through the training set, and the accuracy of the model at this time is obtained through the test set; Next, the corresponding parameters are sorted in descending order of gradient. The first β% of the sorted parameters are then calculated, and it is determined whether (tensor × gradient) is greater than 0. If it is greater than 0, then the b-value of that parameter is... 11 If the bit is equal to 0, or not greater than 0, then the parameter b... 11 The bit is equal to 1; finally, after each layer is processed, the accuracy information at this time is recorded using the test set for comparison with the state before adjustment. If it is better, the state of the current processing layer is switched to the adjusted parameter state. This process is repeated in a loop, and finally the most ideal adaptive parameters are saved.

Images