A traceable deep learning model black-box watermarking scheme

Abstract

The present application relates to the technical field of computer application, specifically to a traceable deep learning model black box watermarking scheme, which proposes a black box watermarking scheme capable of verifying the identity of a deep model. The main method is to generate a trigger set using an image watermarking algorithm to form an efficient, traceable and reproducible deep model black box watermarking scheme, which solves the model identity recognition problem and the current intelligent model watermarking scheme and ownership claim attack problem. The trigger set picture generated by adding random noise or using a deep network is difficult to reproduce. Even in the case of being able to reproduce, it is still impossible to trace and verify the identity. In the case of trigger set loss, the trigger set can still be reproduced through the key and the original trigger picture. This effectively improves the reliability of the scheme.

Description

Technical Field

[0001] This invention relates to the field of computer application technology, specifically to a traceable black-box watermarking scheme for deep learning models. Background Technology

[0002] With breakthroughs in artificial intelligence, significant progress has been made in areas such as image classification and speech recognition. Deep learning, as a representative of AI technology, has already been applied commercially and integrated into people's lives, such as facial recognition and medical image processing. Deep learning models, as the culmination of deep learning technology, require substantial human and material resources for training. For example, the training of the GPT-3 deep learning model in natural language processing cost $12 million. To train a high-performance deep learning model, experts first need to design the model's network structure, and then collect a large number of training samples. After data labeling, these samples serve as training data, requiring significant computational power; training many large-scale models demands staggering amounts of computing power. Furthermore, to obtain a high-performance model, multiple training iterations may be necessary during hyperparameter optimization, incurring substantial time costs (trial and error costs). These factors collectively contribute to the high cost of high-performance models. However, if a meticulously trained high-performance model is copied and misappropriated, it can cause significant economic losses to the model owner. Therefore, how to verify and protect the ownership of deep learning models has become a pressing issue. Existing watermarking schemes for ensuring model security can be broadly divided into two categories: white-box watermarking schemes and black-box watermarking schemes.

[0003] Existing white-box watermarking algorithms have significant drawbacks: limited application scenarios and difficulty in watermark extraction. The prerequisite for using white-box watermarking schemes is access to the complete parameters of the deep learning model during watermark detection and extraction. However, in most cases, deep learning models are deployed on servers, preventing third parties from directly accessing the details of the model parameters; even when access to the intelligent model parameters is possible, understanding the source code is required, making the watermark extraction process complex. Furthermore, most white-box watermarking algorithms are vulnerable to statistical inference attacks. Therefore, many researchers have turned to black-box watermarking. The basic principle of black-box watermarking is to use a set of samples with unique features. Through training, the deep learning model learns the unique features of this sample set. During watermark extraction, ownership of the deep learning model is proven by verifying these unique features. This set of training samples with unique features is also called the trigger set. The unique features of the trigger set are reflected in two aspects: trigger set design and watermark annotation. Trigger set design often involves adding badges or noise to the original image to generate trigger set images. Watermark annotation refers to re-annotating the trigger set, with the new label differing from the original correct label. The design of the trigger set and the annotation together constitute the uniqueness of the model watermark.

[0004] In trigger set design, black-box watermarking schemes often involve adding a logo or random noise to the original image, such as the three watermark image generation mechanisms proposed by Zhang et al. (WM). content WM unrelated WM noise ).

[0005] Existing trigger set designs all have the following problems:

[0006] (1) The features of the trigger set image are highly generalizable. For the trigger set generation mechanism by adding a logo, this feature is highly generalizable. Attackers can easily understand and design similar trigger sets, and therefore may suffer from fraudulent ownership attacks (referring to fraud and claiming ownership by forging similar trigger sets).

[0007] (2) The trigger set lacks traceability. In black-box watermarking schemes, when extracting the watermark to verify the model's identity, the trigger set needs to be input into the server where the target model is deployed through a public port, which may lead to the leakage of the trigger set. Once the trigger set is leaked, its ownership cannot be verified. Attackers can use the stolen trigger set to access the deep learning model through a remote service API, obtain specific annotations of the trigger set, and thus steal the model's intellectual property rights. Existing solutions cannot address the situation of trigger set leakage. Only by achieving traceability of the trigger set can attackers be prevented from using illegally obtained trigger sets to steal the ownership of the model.

[0008] (3) The trigger set, as a key, lacks reproducibility. Trigger sets generated by adding various random noises, manual creation, or GAN-based generation mechanisms cannot be regenerated through trigger set generation mechanisms. Due to the characteristics of deep learning black-box watermarking, the trigger set is an indispensable component for watermark extraction. However, if the trigger set lacks reproducibility, it means that once the trigger set is lost after model training and watermark embedding, even the owner cannot verify the ownership of the deep learning model. Watermarking schemes lacking reproducibility are impractical. Summary of the Invention

[0009] This paper proposes a black-box watermarking scheme capable of verifying the identity of deep models. The main method utilizes an image watermarking algorithm to generate a trigger set, forming an efficient, traceable, and reproducible black-box watermarking scheme for deep models. This solves the model identity recognition problem, addresses current intelligent model watermarking schemes, and mitigates ownership claim attacks.

[0010] The invention is as follows:

[0011] The proposed solution in this invention belongs to the black-box watermarking scheme. This invention includes a watermark embedding algorithm and a watermark extraction algorithm. In the watermark embedding algorithm, a traceable trigger set is generated. Through the model training process, the model learns the features of the trigger set, completing the watermark embedding into the deep learning model. In the watermark extraction algorithm, the trigger set is loaded into the target model via a remote API port. By comparing the output predicted label with the preset label, the watermark is extracted to verify the ownership and identity of the model. Furthermore, by extracting the embedded logo image from the sample images in the trigger set, the identity of the trigger set can be verified, enabling traceability of the trigger set and protecting the ownership of the model.

[0012] A traceable deep learning model black-box watermarking scheme includes an image watermark embedding algorithm, an image watermark extraction algorithm, and a deep learning model watermark embedding algorithm.

[0013] like Figure 1 As shown, the deep learning model watermark embedding algorithm process is as follows:

[0014] Suppose we have an m-classification problem, and the data labels are... It means that, among them, The set of data tags is The watermark embedding algorithm in the black-box watermarking scheme is as follows:

[0015] Input: Training set Watermark length key logo image Threshold Embedding factor ,

[0016] Output: A watermarked deep learning model Trigger set ,in

[0017] 1) From the training set Selected from Sample As the original trigger set For the sample , , It is the corresponding original trigger image. It corresponds to the original tag. ,in That is, the union of all trigger set labels is A subset of a set of labels

[0018] 2) The original trigger set Sample , the original trigger image As a carrier of the original image Each image watermark embedding algorithm is used to embed the logo image. Embedded into each sample image respectively In the middle, generate the corresponding watermark image. ,

[0019] 3) Key It is a newly assigned label sequence For each sample Assign new tags and ensured Its original label They are not the same, and ,

[0020] 4) The watermarked image set generated in step 2 and the preset tags assigned in step 3 together form the trigger set sample set. Among them, the trigger sample , ,

[0021] 5) Trigger set Together with the training set, they serve as training data to train the model network. The resulting watermarked model is the watermarked model. ;

[0022] The image watermark embedding algorithm process is as follows:

[0023] The carrier image is the watermark host medium, and the logo image is the image watermark representing the owner. Both the original image and the logo image are the same size. Image watermarking embedding algorithm is an algorithm that embeds a logo image into a carrier image;

[0024] Input: Original carrier image logo image Threshold Embedding factor ,

[0025] Output: Image with watermark ,

[0026] 1) For carrier images Performing a forward discrete Hadamard transform yields a size of... Hadamard coefficient matrix ,

[0027] 2) Logo image The Middle i Line 1 j The pixel values ​​of the column are denoted as For each pixel value According to the formula Scaling is applied; the scaled matrix pixel values ​​range from [0, ... To satisfy the range of embedded values ​​[0, In [0, Within the range, therefore The theoretical maximum value is ;

[0028] 3) Coefficient matrix The coefficient matrix contains both positive and negative coefficients, so different embedding methods are used for positive and negative coefficients depending on their sign. The Middle i Line 1 j The coefficient of the column is denoted as In the selected embedding region of the coefficient matrix { , },right Perform the following calculations:

[0029]

[0030] In the formula, the Rem function performs the modulo operation;

[0031] 4) For the embedded coefficient matrix Perform the inverse discrete Hadamard transform to obtain the matrix. Then, the matrix values ​​are rounded down to obtain the watermarked image. ;

[0032] like Figure 2 As shown, the image watermark extraction algorithm process is as follows:

[0033] Input: Watermarked image ,parameter and ,

[0034] Output: Extract the logo image ,

[0035] 1) For images with watermarks Performing a forward discrete Hadamard transform yields the corresponding size as coefficient matrix

[0036] 2) Coefficient matrix The Middle i Line 1 j The coefficient of the column is denoted as For the coefficient matrix The selected embedding region { , Coefficient First, perform the absolute value operation: Then from the coefficient value Extract the corresponding values :

[0037]

[0038] In the formula, Represents coefficient value right Perform the modulo operation.

[0039] 3) Extract the values ​​obtained in step 2 , , Using formula The image is then scaled and rounded to obtain the final logo image. ;

[0040] Deep learning model watermark extraction algorithm

[0041] Input: Trigger set key threshold Embedding factor ,

[0042] Output: Watermark extraction rate of the deep learning model, extracted logo image. ,

[0043] 1) Trigger set Load into the model, for each sample The image in Obtain the corresponding output prediction label ;

[0044] 2) Load the preset tags used as keys for each sample. , The output predicted label obtained in step 1 With the preset tag as the key The comparison was performed to count the number of samples whose predicted labels matched the preset labels. ,Right now ;

[0045] 3) The watermark extraction rate is the model's accuracy on the trigger set, i.e. ;

[0046] 4) Trigger set In Sample pictures As a watermarked image The data are input into the image watermark embedding algorithm to obtain the extracted watermark. The extracted watermark images are overlaid to obtain the final extracted logo image. The final logo image will be extracted. Compared to the original logo image By comparing the results, the identity of the trigger set can be proven.

[0047] Beneficial effects of this invention:

[0048] 1. An effective black-box watermarking scheme for proving the identity of deep learning models is proposed.

[0049] The proposed trigger set generation mechanism can effectively generate traceable trigger sets, enabling the embedding of watermarks into deep learning models without sacrificing model accuracy. Watermark extraction can be performed with minimal queries via remote API calls, thereby identifying the owner of the intelligent model. This method requires no access to the specific deep learning model parameters. In practical applications such as forensic evidence collection, the process of technically verifying the model's identity becomes exceptionally simple, requiring only a comparison of the output label classification results with the expected results.

[0050] 2. This solution ensures the traceability and reproducibility of trigger sets, improving the security and reliability of existing black-box watermarking schemes. The proposed solution effectively enhances security and usability. Currently, trigger set images are often created by overlaying an image with specific perturbations. The resulting trigger set, acting as a key, cannot be traced or verified in the event of a leak. Attackers can use stolen trigger sets for fraudulent ownership attacks. Generating trigger set images using an image watermarking mechanism ensures the verification of trigger set identity even if the trigger set is leaked. Even if the trigger set is forged or stolen, ownership can be verified during investigation based on the image watermark information extracted from the trigger set image. Attackers cannot use trigger sets with image watermarks for fraudulent ownership attacks. Furthermore, trigger set images generated by adding random noise or using deep networks are difficult to reproduce. Even if they can be reproduced, traceability and identity verification are impossible. In the event of a lost trigger set, it can still be reproduced using the key and the original trigger image. This effectively improves the reliability of the solution.

[0051] 3. Facilitates the commercialization of deep learning intelligent models and the tracing of sources in case of model leakage.

[0052] This proposed solution facilitates the commercialization of deep learning intelligent models and addresses the issue of model leaks by tracing their origins. Assuming we utilize our deep learning model black-box watermarking scheme, by changing the embedded logo image and key, we can efficiently generate multiple different watermarks, providing each user with a unique watermark. This effectively solves the commercialization problem of deep learning intelligent model watermarking. If a deep learning intelligent model is stolen or leaked, the uniqueness of the watermark helps us quickly locate the source of the leak, thus protecting the legitimate rights of the model owner. Furthermore, if a deep learning model is licensed to multiple users simultaneously, and one user's model is stolen, we can quickly locate that user based on the extracted watermark and the watermark allocation records. Attached Figure Description

[0053] Figure 1. Flowchart of the watermark embedding algorithm for deep learning models.

[0054] Figure 2. Flowchart of the image watermark extraction algorithm. Detailed Implementation

[0055] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0056] This invention provides a technical solution:

[0057] In current black-box watermarking schemes, the key lies in generating a special set of samples called the trigger set. This trigger set is used as part of the training data during the training process, enabling the model to learn not only the basic functionality but also the unique features of the trigger set. During the model's watermark extraction phase, these unique features are used to prove ownership of the model.

[0058] However, current trigger set designs have significant flaws, making it impossible to trace the origin of generated trigger sets and reproduce them if lost. Therefore, we decided to embed image watermarks into trigger sets to address the shortcomings of current black-box watermarking schemes. The distortion introduced by watermark embedding serves as a unique feature of the trigger set, verifying the model's identity during watermark extraction. By embedding watermark information into the trigger set image, ownership of the trigger set is verified. This resolves disputes regarding the identity of the trigger set in cases of its theft.

[0059] Suppose we have an m-classification problem, and the data labels are... It means that, among them, The set of data tags is The proposed black-box watermarking scheme uses the following steps to generate the trigger set:

[0060] 1) Select n samples As the original trigger set For the sample , , It is the corresponding original trigger image. It corresponds to the original tag. .in That is, the union of all trigger set labels is A subset of a set of labels.

[0061] 2) The proposed image watermark embedding algorithm (Algorithm 1) is used on the original trigger set to embed watermark each sample. Trigger set images in As the original image, the logo image The watermark image is embedded into each sample image to generate the corresponding watermark image. .

[0062] 3) For each sample Assign new tags and ensured Its original label They are different, and there are The newly assigned tag sequence will be used as the key.

[0063] 4) The watermarked image set generated in step 2 and the preset tags assigned in step 3 together form the trigger set sample set. Among them, the trigger sample , .

[0064] The process of extracting the model watermark in the black-box watermarking scheme proposed in this invention is as follows:

[0065] 1) Trigger set Load into the model, for each sample The image in Obtain the corresponding output prediction label .

[0066] 2) Load the preset tags used as keys for each sample. , The output predicted label obtained in step 1 With the preset tag as the key Compare them.

[0067] The steps for verifying and tracing the trigger set are as follows:

[0068] 1) Apply the proposed image watermark extraction algorithm (Algorithm 2) to extract the logo image from the trigger set images. .

[0069] 2) Extract the logo image obtained in step 1. Compared to the original logo image By comparing the results, the identity of the trigger set can be proven.

[0070] Pseudocode for watermark embedding algorithm (Algorithm 1):

[0071] The original carrier image used in the embedding algorithm can be either a grayscale image or a color image. For grayscale images, the embedding algorithm can be applied directly; for color images, the embedding algorithm can be applied separately to each of the three channels. The logo image represents the owner's watermark information. Both the original image and the logo image are the same size. .

[0072] Input: Original carrier image logo image Threshold Embedding factor

[0073] Output: Image with watermark

[0074] 1) For carrier images Performing a forward discrete Hadamard transform yields a size of... Hadamard coefficient matrix .

[0075] 2) Logo image The pixel value in the i-th row and j-th column is denoted as For each pixel value According to the formula Scaling is applied. The scaled matrix pixel values ​​range from [0, ... To satisfy the range of embedded values ​​[0, In [0, Within the range, therefore The theoretical maximum value is .

[0076] 3) Coefficient matrix The coefficient matrix contains both positive and negative coefficients, so different embedding methods are used for positive and negative coefficients depending on their sign. The coefficient in the i-th row and j-th column is denoted as In the selected embedding region of the coefficient matrix { , },right Perform the following calculations:

[0077]

[0078] In the formula, the Rem function performs the remainder (rem) operation.

[0079] 4) For the embedded coefficient matrix Perform the inverse discrete Hadamard transform to obtain the matrix. Then, the matrix values ​​are rounded down. This yields the watermarked image. .

[0080] Pseudocode for watermark extraction algorithm (Algorithm 2):

[0081] Input: Watermarked image ,parameter and

[0082] Output: Extract the logo image

[0083] 1) For images with watermarks Performing a forward discrete Hadamard transform yields the corresponding size as coefficient matrix

[0084] 2) Coefficient matrix The Middle i Line 1 j The coefficient of the column is denoted as For the coefficient matrix The selected embedding region { , Coefficient First, perform the absolute value operation: Then from the coefficient values Extract the corresponding values :

[0085]

[0086] In the formula, Represents coefficient value right Perform a modulo operation.

[0087] 3) Extract the values ​​obtained in step 2 , , Using formula The image is then scaled and rounded to obtain the final logo image. .

[0088] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A traceable black-box watermarking scheme for deep learning models, characterized in that: This includes watermark embedding and watermark extraction algorithms. In the watermark embedding algorithm, a traceable trigger set is generated. Through model training, the model learns the features of the trigger set, completing the watermark embedding into the deep learning model. In the watermark extraction algorithm, the trigger set is loaded into the target model via a remote API port. By comparing the output predicted label with the preset label, the watermark is extracted to verify the ownership and identity of the model. The process of watermark embedding algorithm is as follows: Suppose there is Classification problems, data labels It means that, among them, The set of data tags is The watermark embedding algorithm steps are as follows: Input: Training set Watermark length key logo image Other parameters include thresholds. and embedding factors , Output: A watermarked deep learning model Trigger set ,in , 1) From the training set Selected from Sample As the original trigger sample set For the sample , , It is the corresponding original trigger image. It corresponds to the original tag. ,in That is, the union of all trigger set labels is A subset of a set of labels 2) The original trigger set In Sample , the original trigger image As a carrier of the original image Each image watermark embedding algorithm is used to embed the logo image. Embedded into each sample image respectively In the middle, generate the corresponding watermark image. , 3) Re-label the key It is a newly assigned preset tag sequence ,right For each sample Assign new tags and ensured Its original label They are not the same, and The preset tag sequence, which serves as the key, is assigned to each sample one by one. This completes the relabeling of the trigger samples. 4) The watermarked image set generated in step 2, after being re-annotated in step 3, forms the trigger set sample set. Among them, the trigger sample , , 5) Trigger set and training set Using these datasets together as training data, the model network is trained, and the resulting model is the watermarked deep learning model. ; The image watermark embedding algorithm process is as follows: The carrier image is the watermark host medium, and the logo image is an image watermark representing the owner's information. Both the original image and the logo image are the same size. Image watermarking embedding algorithm is an algorithm that embeds a logo image into a carrier image; Input: Original carrier image logo image Threshold Embedding factor , Output: Image with watermark , 1) For carrier images Performing a forward discrete Hadamard transform yields a size of... Hadamard coefficient matrix , 2) Logo image The Middle i Line 1 j The pixel values ​​of the column are denoted as For each pixel value According to the formula Scaling is applied; the scaled matrix pixel values ​​range from [0, ... To satisfy the range of embedded values ​​[0, In [0, Within the range, therefore The theoretical maximum value is ; 3) Coefficient matrix The coefficient matrix contains both positive and negative coefficients, so different embedding methods are used for positive and negative coefficients depending on their sign. The Middle i Line 1 j The coefficient of the column is denoted as In the selected embedding region of the coefficient matrix { , },right Perform the following calculations: In the formula, the Rem function performs the modulo operation; 4) For the embedded coefficient matrix Perform the inverse discrete Hadamard transform to obtain the matrix. Then, the matrix values ​​are rounded down to obtain the watermarked image. ; Watermark extraction algorithms are used to verify ownership of target models. The process of watermark extraction algorithms is as follows: Input: Trigger set key threshold Embedding factor , Output: Watermark extraction rate of the deep learning model, extracted logo image. , 1) Trigger set Loaded into the target model via the service API port, Each sample The image in Obtain the corresponding output prediction label ; 2) Load the preset tags used as keys for each sample. , The output predicted label obtained in step 1 With the preset tag as the key The comparison was performed to count the number of samples whose predicted labels matched the preset labels. ,Right now The watermark extraction rate of a deep learning model is the model's accuracy on the trigger set, i.e. ; 3) Trigger set In Sample pictures As a watermarked image The images are input into the image watermark extraction algorithm to obtain the extracted logo image. The extracted logo images are overlaid and averaged to obtain the final extracted logo image. Logo images were extracted through comparison. Compared to the original logo image This proves the identity of the trigger set and enables tracing the origin of the trigger set. The image watermark extraction algorithm process is as follows: Input: Watermarked image ,parameter and , Output: Extract the logo image , 1) For images with watermarks Performing a forward discrete Hadamard transform yields the corresponding size as coefficient matrix , 2) Coefficient matrix The Middle i Line 1 j The coefficient of the column is denoted as For the coefficient matrix The selected embedding region { , Coefficient First, perform the absolute value operation: Then from the coefficient value Extract the corresponding values : In the formula, Represents coefficient value right Perform the modulo operation. 3) Extract the values ​​obtained in step 2 , , Using formula The image is then scaled and rounded to obtain the final logo image. .

Images