Deep learning model watermarking method fusing iris information and random samples

Abstract

The present application relates to a deep learning model watermark protection method fusing iris information and random samples, and belongs to the field of computer and information science. First, the iris information of the model owner is extracted, and three groups of random samples are obtained by randomly sampling the model training set three times. Second, a watermark sample is generated by using the Huffman tree algorithm combined with the owner's iris information and the first group of random samples. Third, a trigger sample is obtained by fusing the watermark sample and the second group of random samples by using the non-subsampled shearlet transform and the Laplacian pyramid algorithm. Finally, the trigger sample and the third group of random samples are merged to form a new data set, and the original model is trained using the data set to obtain a fine-tuned deep neural network watermark model. The present application proposes a deep learning model watermark protection method fusing iris information and random samples to solve the problem of fixed trigger sample selection, and improves the attack resistance of the model watermark.

Description

Technical Field

[0001] This invention relates to a watermarking protection method for deep learning models that integrate iris information and random samples, belonging to the field of computer and information science and technology. Background Technology

[0002] As deep neural networks play an increasingly important role in many fields, the security issues faced by deep learning models have also received widespread attention. Successfully training a deep learning model typically requires enormous human and material resources, which incentivizes malicious adversaries to steal models and conceal their activities, leading to infringement of intellectual property rights and potential economic losses. Recently, researchers have introduced multimedia content protection methods into the field of deep learning model protection. Leveraging the overfitting characteristic of deep learning models, they embed watermarks during model training to hide model information without affecting the integrity and readability of the original information, effectively preventing potential attacks and protecting the intellectual property rights of deep learning models. Therefore, researching an effective watermarking protection method has significant theoretical and practical value for protecting the privacy and security of deep learning models.

[0003] Existing watermark embedding methods mainly fall into two categories: First, embedding within the network itself. This method uses internal network information as a carrier, adjusting the model structure by embedding model weights, new layers, or network outputs to embed the watermark from the perspective of the model's internal structure. However, since this internal structure information needs to be publicly disclosed, attackers can easily exploit it. Second, establishing a network backdoor. This method primarily targets image classification tasks. By embedding a backdoor into the deep learning model, a special input-output relationship is introduced. When verifying model ownership, the owner can use the trigger samples embedded as the backdoor to generate a trigger set. Since only the owner knows the backdoor, ownership can be proven by demonstrating the input-output relationship. However, existing trigger set selection methods are relatively fixed and cannot be arbitrarily changed, resulting in poor robustness of the model watermark.

[0004] After obtaining the watermark model, when a suspicious model is detected, it is necessary to verify whether the model is a derivative of the original model. Currently, the mainstream verification method is to check whether the outputs of the two models under the same input follow the same distribution, thereby determining the similarity between the models. However, this method does not quantify the similarity between models to a specific numerical value, and its accuracy needs further improvement.

[0005] In summary, facing complex and ever-changing attack scenarios, current watermark embedding methods require exposing the internal structure of the model when embedding within the network, making effective protection difficult. Furthermore, methods that establish network backdoors rely on fixed trigger set selection. Existing methods also lack specific quantification for determining the similarity between models. Therefore, this invention proposes a deep learning model watermark protection method that integrates iris information and random samples. Summary of the Invention

[0006] The purpose of this invention is to address the problem of fixed selection methods for existing trigger samples by proposing a deep learning model watermark protection method that integrates iris information and random samples. By randomly sampling the model training set three times for watermark embedding, the watermark's resistance to attacks is improved. Furthermore, to address the lack of specific quantitative indicators for judging similarity between existing models, this invention proposes a method that uses distributional similarity and linear similarity to calculate model similarity and determine whether a suspicious model is similar to the original model.

[0007] The design principle of this invention is as follows: First, the owner's iris information is extracted from the owner's iris image, and three sets of random samples are randomly sampled three times from the model training set. Second, the Huffman tree algorithm is used to combine the owner's iris information and the first set of random samples to generate watermark samples. Then, the non-subsampled shear wave transform and Laplace pyramid algorithm are used to fuse the watermark samples and the second set of random samples to obtain trigger samples. Finally, the trigger samples are merged with the third set of random samples to form a new dataset. The original model is trained using this dataset to obtain a fine-tuned deep neural network watermark model, and the model is verified by combining distribution similarity and linear similarity.

[0008] The technical solution of the present invention is achieved through the following steps:

[0009] Step 1: Extract the owner's iris information from the owner's iris image, and randomly sample the model training set three times to obtain three sets of random samples.

[0010] Step 1.1: Preprocess the owner's iris image, including iris localization, iris image normalization, and iris image enhancement.

[0011] Step 1.2: Extract and encode the texture information of the owner's iris.

[0012] Step 1.3: Randomly sample the model training set three times to obtain three sets of random samples.

[0013] Step 2: Generate a watermark sample by combining the owner's iris information and a random sample.

[0014] Step 2.1: Encrypt the owner's iris information obtained in Step 1.2 and the first set of random samples obtained in Step 1.3 to obtain the watermark sample.

[0015] Step 3: Merge the watermark sample and the random sample to obtain the trigger sample.

[0016] Step 3.1: Use non-subsampled shear wave transform (NSST) to divide the second set of random samples obtained in step 1.3 into low-frequency subband and high-frequency subband.

[0017] Step 3.2: The watermark sample is fused with the low-frequency subband portion obtained in Step 3.1 using the Laplace pyramid algorithm (LP).

[0018] Step 3.3: The watermark sample is fused with the high-frequency subband portion obtained in step 3.1 using a simplified pulse-coupled neural network (PCNN).

[0019] Step 3.4: Combine the results of Step 3.2 and Step 3.3 to obtain the trigger sample.

[0020] Step 4: Combine the trigger samples obtained in Step 3.4 with the third set of random samples obtained in Step 1.3 to form a new training set, generate a fine-tuned deep neural network watermark model, and use the watermark to verify the model.

[0021] Step 4.1: Merge the trigger sample with the third set of random samples to form a new dataset, and use this dataset to train the original model to obtain the fine-tuned deep neural network watermarking model.

[0022] Step 4.2: Calculate the distribution similarity between the model to be detected and the original model.

[0023] Step 4.3: Calculate the linear similarity between the model to be detected and the original model.

[0024] Step 4.4: Calculate the model similarity between the model to be detected and the original model.

[0025] Step 4.5: Use model similarity to determine whether the model to be detected is a derivative model of the original model.

[0026] Beneficial effects

[0027] Compared to existing deep learning model watermarking techniques, this invention improves the watermark's resistance to attacks by adding randomization during iris information processing, watermark sample generation, and trigger sample generation by randomly sampling the model training dataset three times. Simultaneously, it combines distributional similarity and linear similarity to calculate model similarity, thus improving the accuracy of similarity judgment between models. Attached Figure Description

[0028] Figure 1 This is a schematic diagram of the watermark protection method of the deep learning model that integrates iris information and random samples according to the present invention.

[0029] Figure 2 This is a flowchart illustrating the watermark verification process of the deep learning model-based watermark protection method that integrates iris information and random samples, as described in this invention. Detailed Implementation

[0030] To better illustrate the purpose and advantages of the present invention, the implementation methods of the present invention will be further described in detail below with reference to examples.

[0031] The experimental data consisted of the model's training dataset and the owner's iris image. Four attack methods—removal attack, theft attack, escape attack, and ambiguity attack—were used to attack the original unwatermarked model, resulting in four suspicious models. These suspicious models were uniformly represented using DM to verify the robustness of the watermarked model.

[0032] This experiment was conducted on a computer with the following configuration: 12th Gen Intel(R) Core(TM) i9-12900H, CPU clock speed 2.50GHz, 32GB RAM, and the operating system is 64-bit Windows 11.

[0033] The specific process is as follows:

[0034] Step 1: Extract the owner's iris information from the owner's iris image, and randomly sample the model training set three times to obtain three sets of random samples.

[0035] Step 1.1: Preprocess the owner's iris image, including iris localization, iris image normalization, and iris image enhancement.

[0036] Step 1.1.1: Use Hough transform to remove the interference parts in the image and retain the iris part of the image.

[0037] Step 1.1.2: Adjust the extracted iris images to the same size and position to eliminate the effects of translation, scaling, etc. After normalization, the iris image size is 256×64, and it is decomposed into 16×16 sub-images.

[0038] Step 1.1.3: Histogram equalization is used to process the iris image to improve the contrast of the image texture.

[0039] Step 1.2: Utilizing the directionality of the Gabor filter, the preprocessed 16×16 sub-image is convolved through an even / odd filter to obtain a 2048-bit iris texture information code.

[0040] Step 1.3: Randomly sample the model training dataset three times to obtain three sets of random samples D. train-sample D train_merge and D train_30％The first and second sets of random samples each have a size of 10% of the model training dataset, while the third set of random samples has a size of 30% of the model training dataset.

[0041] Step 2: Generate a watermark sample by combining the owner's iris information and a random sample.

[0042] Step 2.1: Encode the owner's iris texture information obtained in Step 1.2 and the first set of random samples obtained in Step 2.1 using a Huffman tree to generate a watermark sample. At this time, the watermark sample has the same structure as the first set of random samples, that is, the data capacity in the watermark sample is 10% of the model training dataset.

[0043] D key =f Hufman (code, D) train-sample (1)

[0044] Step 3: Merge the watermark sample and the second set of random samples to obtain the trigger sample.

[0045] Step 3.1: Process the second set of random samples using Non-Subsampled Shear Wave Transform (NSST) to obtain the low-frequency subband D. low and high-frequency subband D high .

[0046] Step 3.2: The watermark sample is fused with the low-frequency sub-band portion of the second set of random samples using the Laplacian pyramid algorithm (LP) to make the image feature information more apparent. LP This is the result after fusion.

[0047] D LP =f LP (D low D key (2)

[0048] Step 3.3 involves fusing the watermarked sample with the high-frequency subband portion of the second set of random samples using a simplified pulse-coupled neural network (PCNN), finding the optimal parameter values, and reconstructing the fused image. The resulting image has high resolution and clarity. PCNN This is the result after fusion.

[0049] D PCNN =f PCNN (D high D key (3)

[0050] Step 3.4: Combine the results of steps 3.2 and 3.3 to obtain the trigger sample D. trigger .

[0051] Dtrigger =Merge(D LP D PCNN (4)

[0052] Step 4: Combine the trigger samples and random samples to form a new training set, generate a fine-tuned deep neural network watermarking model M1, and validate the model.

[0053] Step 4.1: Merge the trigger samples obtained in Step 3.4 with the third set of random samples to form a new dataset D. train-new The model was retrained to generate a fine-tuned deep neural network watermarking model M1.

[0054] D train-new =Merge(D train_30％ D trigger (5)

[0055] M1 = TrainModel(D train-new (6)

[0056] Step 4.2, calculate the distribution similarity S between the model to be detected DM and the original model M. dis .

[0057] Step 4.2.1: Input the same original dataset D into both model M and model DM. train Let the output of the two models be denoted as result. M1 and result DM .

[0058] Step 4.2.2: Calculate the parameter p used to determine the statistical significance in the hypothesis test using the Kolmogorov-Smirnov (KS) test, and test the result. M1 and result DM Whether they follow the same distribution. If p≤0.05, the two models are considered to follow the same distribution, and S dis =1, otherwise S dis =0.

[0059]

[0060] Step 4.3: Establish a mapping relationship between the outputs of the two models under the same input, compare the changes in the mapping relationship under multiple inputs, and calculate the linear similarity S. Linear .

[0061] Step 4.3.1, process the original dataset D train The data is divided into 10 groups, each containing the original dataset D. train10% of the data was input into model M and model DM respectively, resulting in 10 sets of output results for the two models. Each set of output results was recorded as a result. M1i and result DMi , where 1≤i≤10.

[0062] Step 4.3.2: Establish a linear regression mapping relationship between the outputs of model M and model DM in each group when they have the same input. DMi =a i result M1i +b i 10 groups of a were calculated from the data in each group. i The value of .

[0063] Step 4.3.3, calculate a in each group i The linear similarity S is calculated using hyperparameter optimization based on the residuals e and standard deviations s. Linear Here, α is a hyperparameter that can be adjusted according to the actual dataset used; Let a be the mean of a.

[0064] S Linear =αe+(1-α)s (8)

[0065]

[0066]

[0067] Step 4.4, using distribution similarity S dis And linear similarity S Linear Calculate the model similarity S.

[0068] S = S dis S Linear (11)

[0069] Step 4.5 uses model similarity to determine whether the model to be tested is a derivative model of the original model. The larger the value of S, the closer the decision boundaries of the two models are, and the more similar the two models are. The threshold l is a judgment threshold selected through numerous experiments. If the model similarity S is greater than the threshold l, then DM and M1 are considered to have similar decision boundaries, and the model to be tested is a derivative model of the original model; if the model similarity S is less than the threshold l, then the model to be tested is considered unrelated to the original model.

[0070] Experimental results show that this invention improves the accuracy of data by quantifying the similarity between models. Compared with the current high-level method EWE, this invention solves the problem that EWE cannot effectively deal with ambiguity attacks, and improves the robustness of model watermarks under three attack methods: removal attack, theft attack, and escape attack.

[0071] The above detailed description further illustrates the purpose, technical solution, and beneficial effects of the invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A watermarking protection method based on a deep learning model that integrates iris information and random samples, characterized in that... The method includes the following steps: Step 1: Extract the owner's iris information from the owner's iris image, and randomly sample the model training set three times to obtain three sets of random samples; Step 2: Use the Huffman tree algorithm to combine the owner's iris information and the first set of random samples to generate a watermark sample; Step 3: Then, using the non-subsampled shear wave transform and the Laplace pyramid algorithm, the watermark sample and the second set of random samples are fused to obtain the trigger sample; Step 4: Merge the trigger sample with the third set of random samples to form a new dataset. ,use The original model was trained to obtain a fine-tuned deep neural network watermarking model. The model was validated by combining distributional similarity and linear similarity, and the output similarity was calculated using the linear regression mapping relationship between the outputs of the two models under the same input. Combining distribution similarity and linear similarity Calculate model similarity And utilize model similarity The determination is made as to whether the model to be detected is a derivative model of the original model, wherein... The suspicious model obtained after attacking the original model without watermarking. and For the first The outputs of two models with the same input. and For the first The linear regression mapping parameters between the outputs of two models with the same input. for The residual, for standard deviation for The mean, These are hyperparameters and can be adjusted based on the actual dataset used. These are parameters in the KS test, used to determine statistical significance in hypothesis testing; Distribution similarity is used to record whether the outputs of the two models conform to the same distribution, where The larger the threshold value, the closer the decision boundaries of the two models are, and the more similar the two models are. The judgment threshold was selected through numerous experiments. If the model similarity... Greater than the threshold Then the suspicious model is identified. With watermark model If the decision boundaries are similar, the model to be detected is a derivative model of the original model; if the model similarity is... Less than the threshold Then it is determined that the model to be tested is unrelated to the original model; （1） （2） （3） （4） （5） （6）。 2. The watermarking protection method for deep learning models that integrate iris information and random samples according to claim 1, characterized in that: In step 2, random samples were added when encrypting the owner's iris information using a Huffman tree. If the owner's iris information is leaked, the model watermark can be protected from being easily extracted by replacing the first set of random samples, thus improving the robustness of the model watermark.

3. The watermarking protection method for deep learning models that integrate iris information and random samples according to claim 1, characterized in that: In step 3, after obtaining low-frequency and high-frequency subbands from the second group of random samples using non-subsampled shear wave transform, for the low-frequency subband, the images of the corresponding frequency bands are fused by weighted averaging according to the Laplacian pyramid algorithm to make the image feature information more obvious; for the high-frequency subband, a simplified pulse-coupled neural network model is used for fusion to find the optimal parameter values ​​and reconstruct a fused image with high resolution and clarity.

Images