Model watermark embedding method based on forgetting early warning, electronic equipment and medium

Abstract

The invention discloses a forgetting early warning-based model watermark embedding method, electronic equipment and a medium. The method comprises the following steps of selecting an original image data set; generating a watermark sample to be embedded; randomly selecting a part of image samples from the original image data set, determining a watermark position through an attention heat map technology, embedding the watermark sample into the selected image sample, and calculating a target category corresponding to the watermark image sample to obtain a watermark image data set; alternately training a classification model by using the original image data set and the watermark image data set; the trained classification model is used for realizing image classification and identification and realizing watermark traceability.

Description

Technical Field

[0001] The present invention belongs to the technical field of model property protection, and in particular relates to a model watermark embedding method, electronic equipment, and medium based on forgetting warning. Background Art

[0002] With the rapid development of computer technology, artificial intelligence technologies such as deep learning have achieved great success in computer vision, pattern recognition, and natural language processing, which has prompted technology giants such as Google and Microsoft to apply them to commercial products, improving service quality and economic benefits. However, building excellent artificial intelligence models requires a lot of data, computing resources, and expertise. At the same time, artificial intelligence models are vulnerable to theft, tampering, and trafficking by unauthorized users. As artificial intelligence develops rapidly, how to protect the intellectual property rights of these models has become an important issue of concern to academia and industry.

[0003] The most common technology for protecting the intellectual property rights of intelligent models is watermark embedding based on backdoors. This is a covert digital watermarking technology that uses the backdoor mechanism of the artificial intelligence model to embed watermarks to track and authenticate the model. In this technology, the watermark is embedded in the weight parameters or input data of the model, but it has nothing to do with the main task of the model, so it will not affect the normal performance of the model. When the backdoor is activated, the watermark information can be activated and extracted to prove the ownership of the model or track it. However, most of the existing backdoor-based watermark embedding technologies directly embed into the image in the form of randomly generated perturbations, which lack semantic information and have insignificant features. They are difficult to extract and verify, and directly embedding semantic-free labels or text labels will affect the performance of the model's main task to a certain extent. In addition, the existing backdoor-based watermark embedding technology usually randomly selects a category or creates a new category as a category label, which will cause the embedded watermark to be easily removed in subsequent work and is unstable. In addition, the existing watermark embedding technology is not robust enough against common attack methods such as pruning and quantization. Finally, the existing model watermark de-embedding technology usually cannot extract the initial watermark during the verification process, but is determined by specific activation features or model outputs. This may have errors to a certain extent and cannot truly guarantee the property rights security of the model. Summary of the invention

[0004] In order to solve the problems in the existing backdoor-based watermark embedding technology that the watermark lacks semantic information, affects the performance of the main task of the model and is easily eliminated by means of pruning, quantization, etc., and the target class is randomly selected or a new class is created, resulting in insufficient stability of the watermark information in the subsequent development of the model, the present invention provides a model watermark embedding method, electronic device, and medium based on forgetting warning.

[0005] In a first aspect, an embodiment of the present invention provides a model watermark embedding method based on forgetting warning, the method comprising the following steps:

[0006] Select the original image dataset;

[0007] Generate a watermark sample to be embedded;

[0008] Randomly select some image samples from the original image dataset, determine the watermark position through the attention heat map technology, embed the watermark sample into the selected image sample, and calculate the target category corresponding to the watermark image sample to obtain the watermark image dataset;

[0009] The classification model is trained alternately using the original image data set and the watermark image data set; the trained classification model is used to realize image classification and recognition, and can also realize watermark traceability.

[0010] Furthermore, the process of generating a watermark sample to be embedded includes:

[0011] Get the watermark information to be embedded;

[0012] The watermark information is encoded using ASCII to obtain a two-dimensional matrix as the watermark sample to be embedded.

[0013] Furthermore, the watermark information includes but is not limited to date and unit name, and is used to characterize the classification model.

[0014] Furthermore, some image samples are randomly selected from the original image data set, and the watermark position is determined by the attention heat map technology. The process of embedding the watermark sample into the selected image sample includes:

[0015] Input the image sample into the convolutional neural network, and output the attention heat map corresponding to each image sample through the gradient weighted class activation mapping technology, wherein the attention heat map is used to illustrate the key areas in the image sample that affect the prediction of the convolutional neural network;

[0016] Embed the watermark sample into the key area of ​​the image sample; specifically, align the position of each element in the binary matrix with the position of the pixel in the image sample, and multiply the element value with the pixel value to obtain a new pixel value to achieve watermark embedding;

[0017] Furthermore, the expression of the gradient weighted class activation mapping technique is as follows:

[0018]

[0019] In the formula, A represents a feature layer. In this example, it refers to the feature layer output by the last convolutional layer; k represents the kth channel in feature layer A; c represents category c; A k Represents the data of channel k in feature layer A; Represents A k The weight of

[0020] The calculation formula is as follows:

[0021]

[0022] In the formula, y c Represents the score predicted by the convolutional neural network for category c without softmax activation; Represents the data of feature layer A at the coordinate (i, j) in channel k; Z is equal to the product of the width and height of the feature layer.

[0023] Furthermore, the process of calculating the target category corresponding to the watermark image sample includes:

[0024] The forgetting index of each category is calculated according to the forgetting warning index, and the category with the highest forgetting index is selected as the target category corresponding to the watermark image sample. The watermark image sample is placed in the target category to obtain the watermark image data set.

[0025] Among them, the calculation formula of the forgetting warning indicator is as follows:

[0026]

[0027] Where T is the number of categories, α base,i is the test accuracy of the first classification task (base set) after learning the i-th new classification task, α ideal is the accuracy of the offline classification model on the base set.

[0028] Furthermore, the method further comprises: extracting and verifying watermarks from the trained classification model;

[0029] Specifically, it includes:

[0030] Input the watermark image dataset into the trained classification model to obtain the output category corresponding to each watermark image sample;

[0031] According to the output category corresponding to each watermark image sample, the watermark image samples that do not match the true label are eliminated;

[0032] Calculate the classification confidence distribution of the filtered watermark image samples after inputting them into the trained classification model, and select the watermark image sample with the highest classification confidence;

[0033] The two-dimensional matrix in the watermark image sample is extracted and inversely transformed to obtain text information, thereby completing the watermark extraction and verification, and realizing the property rights protection of the model.

[0034] In a second aspect, an embodiment of the present invention provides an electronic device, comprising a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the above-mentioned model watermark embedding method based on forgetting warning.

[0035] In a third aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, and when the program is executed by a processor, the above-mentioned model watermark embedding method based on forgetting warning is implemented.

[0036] In a fourth aspect, an embodiment of the present invention provides a computer program product, including a computer program / instruction, which implements the above-mentioned model watermark embedding method based on forgetting warning when executed by a processor.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] The present invention provides a model watermark embedding method based on forgetting warning, which converts the text label information representing the model into a binary data conversion binary matrix, and uses this as watermark information to embed into the image; then, the forgetting tendency degree of the current model for all categories of data is obtained by using the forgetting warning method, and the least likely to be forgotten category is selected as the target label, and the watermark embedding position is determined by the attention heat map, covering the key areas originally concerned by the model, ensuring the performance of the main task of the model, and obtaining a watermark image data set embedded with the watermark; the original image data set and the watermark image data set are used to alternately train the classification model; the trained classification model is used to realize image classification and recognition, and can realize watermark tracing. Finally, the watermark information embedded in the model is extracted and restored, and its restoration degree is evaluated to realize the property rights verification of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0040] Figure 1 A schematic diagram of the flow of a model watermark embedding method based on forgetting warning provided by an embodiment of the present invention;

[0041] Figure 2 A schematic diagram of generating a binary matrix for watermark text information provided by an embodiment of the present invention;

[0042] Figure 3 A schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] It should be noted that, in the absence of conflict, the features in the following embodiments and implementations may be combined with each other.

[0045] like Figure 1 and Figure 2 As shown, an embodiment of the present invention provides a model watermark embedding method based on forgetting warning, the method comprising:

[0046] Step S1, selecting an original image data set.

[0047] Exemplarily, in this example, the original image dataset CIFAR10 dataset is selected; the CIFAR10 dataset has a total of 60,000 samples, each of which is a 32*32 pixel RGB image (color image) with 10 categories of objects, and the label values ​​are distinguished by 0 to 9, which are airplane, automobile, bird, cat, deer, dog, frog, horse, ship and truck.

[0048] Furthermore, the original image dataset is analyzed and preprocessed to match pixel images with their attribute labels one by one; and the original image dataset is divided into a training sample set and a test sample set. In this example, the original image dataset is divided into a training dataset of 50,000 images and a test dataset of 10,000 images.

[0049] Step S2, generating watermark samples to be embedded; randomly selecting some image samples from the original image data set, determining the watermark position through the attention heat map technology, embedding the watermark samples into the selected image samples, and calculating the target category corresponding to the watermark image samples to obtain the watermark image data set.

[0050] Specifically, step S2 includes the following sub-steps:

[0051] Step S201, generating a watermark sample to be embedded, including:

[0052] The watermark information to be embedded is obtained, where the watermark information includes but is not limited to date and unit name, and is used to characterize the model.

[0053] The watermark information is encoded using ASCII (American Standard Code for Information Interchange) to obtain a two-dimensional matrix as a watermark sample to be embedded.

[0054] Exemplarily, in this example, "2024ZJUT" is used as the watermark information to be embedded. The number "2" in "2024ZJUT" can be expressed as "00000010", the character "Z" can be expressed as "01011010", the number "0" can be expressed as "00000000", the number "4" can be expressed as "00000100", the character "J" can be expressed as "01001010", the character "U" can be expressed as "01010101", and the character "T" can be expressed as "01010100". The final watermark information is converted into an 8×8 binary matrix as a watermark sample to be embedded.

[0055] Step S202, randomly selecting some image samples from the original image data set, determining the watermark position through the gradient weighted class activation mapping technology, and embedding the watermark sample into the selected image sample;

[0056] Specifically, the image samples are input into the Convolutional Neural Networks (CNN), and the attention heat map corresponding to each image sample is output through the Gradient Weighted Class Activation Mapping (Grad-CAM) technology. The attention heat map is used to illustrate the key areas in the image samples that affect the prediction of the convolutional neural network, and the watermark samples are embedded in the key areas of the image samples. Specifically, the position of each element in the binary matrix is ​​aligned with the position of the pixel point in the image sample, and the element value is multiplied by the pixel value to obtain a new pixel value to achieve watermark embedding.

[0057] Further, the working principle of Gradient Weighted Class Activation Mapping (Grad-CAM) is as follows:

[0058] Forward propagation: The input image is passed through the CNN to generate feature maps at various convolutional layers.

[0059] Compute gradients: Compute the gradient of the target class (the class being predicted) with respect to the feature map of the last convolutional layer.

[0060] Global Average Pooling (GAP): The gradients are then globally averaged to provide an importance score for each feature map. This step captures the importance of each feature map to the target class.

[0061] Weighted sum of activation maps: The importance scores obtained in the previous step are used as weights to compute the weighted sum of activation maps of the last convolutional layer.

[0062] ReLU and Upsampling: The resulting map is passed through a ReLU activation function to remove any negative values ​​and then upsampled to the size of the input image.

[0063] Visualization: Overlay the upsampled heatmap onto the input image to visualize the regions in the image that are important for the CNN prediction.

[0064] Furthermore, the expression of Grad-CAM is as follows:

[0065]

[0066] In the formula, A represents a feature layer. In this example, it refers to the feature layer output by the last convolutional layer; k represents the kth channel in feature layer A; c represents category c; A k Represents the data of channel k in feature layer A; Represents A k The weight of .

[0067] The calculation formula is as follows:

[0068]

[0069] In the formula, y c Represents the score predicted by the convolutional neural network for category c without softmax activation; Represents the data of feature layer A at the coordinate (i, j) in channel k; Z is equal to the product of the width and height of the feature layer.

[0070] Step S203, calculating the target category corresponding to the watermark image sample;

[0071] Specifically, the forgetting index of each category is calculated according to the forgetting warning index, and the category with the highest forgetting index is selected as the target category corresponding to the watermark image sample. The watermark image sample is placed in the target category to obtain the watermark image data set.

[0072] Specifically, the calculation formula of the forgetting warning indicator is as follows:

[0073]

[0074] Where T is the number of categories, α base,i is the test accuracy of the first classification task (base set) after learning the i-th new classification task, α ideal is the accuracy of the offline classification model on the base set.

[0075] Step S3, alternately train the classification model using the original image data set and the watermark image data set; the trained classification model is used to implement image classification and recognition, and can realize watermark tracing.

[0076] Specifically, in this example, a three-layer deep neural network framework with 64 input nodes, 10 output nodes, and 32 hidden nodes in the middle layer is built. The ReLU activation function is used to increase the nonlinearity of the neural network model. Its expression is as follows:

[0077] f(x)=max(0,x)

[0078] A Dropout with a probability of 0.8 is set after the output of each layer. In each generation of training, there is a probability of 0.2 to actively and temporarily ignore some hidden nodes to speed up the training of the classification model and prevent overfitting.

[0079] The final result is output by the softmax function, and its expression is as follows:

[0080]

[0081] Among them, z i is the output value of the ith node, C is the number of output probabilities, that is, the number of categories, which is 10 categories in this example. The output values ​​of the 10 categories are converted into a probability distribution in the range of [0, 1] and 1 through the Softmax function to obtain the prediction result.

[0082] The loss value between the predicted value obtained by forward propagation and the true target label is then calculated based on the cross entropy loss function, which is expressed as follows:

[0083]

[0084] Where M represents the number of categories, y ic is a sign function (0 or 1), if the true category of sample i is equal to c, it takes 1, otherwise it takes 0, p ic Indicates the probability that the observed sample i belongs to category c. Next, in the back-propagation process, the partial derivative of the loss function L with respect to the model parameters is calculated to obtain a gradient value, and then the momentum-based stochastic gradient descent method SGD with a learning rate of 0.1 and a momentum factor of 0.5 is used to update the model parameters. The expression is as follows:

[0085] v i =αv i-1 +η▽L(θ)

[0086] θ i =θ i-1 -v i

[0087] Among them, α represents the momentum parameter, η is the learning rate, ▽L(θ) is the gradient value obtained by taking the partial derivative of the loss function with respect to the parameter θ, and θ i-1 and θ i Represent the model parameters before and after the update respectively.

[0088] Step S4, extracting and verifying the watermark of the trained classification model.

[0089] Specifically, the watermark image dataset is input into the trained classification model to obtain the output category corresponding to each watermark image sample;

[0090] According to the output category corresponding to each watermark image sample, the watermark image samples that do not match the true label are eliminated;

[0091] Calculate the classification confidence distribution of the filtered watermark image samples after inputting them into the trained classification model, and select the watermark image sample with the highest classification confidence;

[0092] The two-dimensional matrix in the watermark image sample is extracted and inversely transformed to obtain text information, thereby completing the watermark extraction and verification, and realizing the property rights protection of the model.

[0093] In summary, the present invention provides a model watermark embedding method based on forgetting warning, which converts the text label information representing the model into a binary data conversion binary matrix, and uses this as watermark information to embed into the image; then, the forgetting tendency of the current model for all categories of data is obtained by using the forgetting warning method, and the category that is least likely to be forgotten is selected as the target label to obtain a watermark image data set embedded with the watermark; the classification model is alternately trained using the original image data set and the watermark image data set; the trained classification model is used to realize image classification and recognition, and can realize watermark tracing. Finally, the watermark information embedded in the model is extracted and restored, and its restoration degree is evaluated to realize the property rights verification of the model.

[0094] Accordingly, the present application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned model watermark embedding method based on forgetting warning. Figure 3 As shown, a hardware structure diagram of any device with data processing capability in which the model watermark embedding method based on forgetting warning provided by an embodiment of the present invention is located, except Figure 3 In addition to the processor, memory and network interface shown, any device with data processing capability in which the apparatus in the embodiment is located may also include other hardware, generally based on the actual functions of the device with data processing capability, which will not be described in detail.

[0095] Correspondingly, the present application also provides a computer-readable storage medium on which computer instructions are stored, and when the instructions are executed by the processor, the model watermark embedding method based on forgetting warning as described above is implemented. The computer-readable storage medium can be an internal storage unit of any device with data processing capabilities described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium can also be an external storage device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), an SD card, a flash card (Flash Card), etc. equipped on the device. Further, the computer-readable storage medium can also include both an internal storage unit of any device with data processing capabilities and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and can also be used to temporarily store data that has been output or is to be output.

[0096] Those skilled in the art will readily appreciate other embodiments of the present application after considering the description and practicing the contents disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary technical means in the art that are not disclosed in the present application. The description and examples are intended to be exemplary only.

[0097] It should be understood that the present application is not limited to the exact construction that has been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof.

Claims

1. A model watermark embedding method based on forgetting warning, characterized in that: The method comprises the following steps: Select the original image dataset; Generate a watermark sample to be embedded; Randomly select some image samples from the original image dataset, determine the watermark position through the attention heat map technology, embed the watermark sample into the selected image sample, and calculate the target category corresponding to the watermark image sample to obtain the watermark image dataset; The classification model is trained alternately using the original image data set and the watermark image data set; the trained classification model is used to realize image classification and recognition, and can also realize watermark traceability.

2. According to the model watermark embedding method based on forgetting warning according to claim 1, it is characterized in that: The process of generating the watermark sample to be embedded includes: Get the watermark information to be embedded; The watermark information is encoded using ASCII to obtain a two-dimensional matrix as the watermark sample to be embedded.

3. According to the model watermark embedding method based on forgetting warning according to claim 2, it is characterized in that: The watermark information includes but is not limited to date and unit name, and is used to characterize the classification model.

4. The model watermark embedding method based on forgetting warning according to claim 1 is characterized in that: The process of randomly selecting some image samples from the original image dataset, determining the watermark position through the attention heat map technology, and embedding the watermark sample into the selected image sample includes: Input the image sample into the convolutional neural network, and output the attention heat map corresponding to each image sample through the gradient weighted class activation mapping technology, wherein the attention heat map is used to illustrate the key areas in the image sample that affect the prediction of the convolutional neural network; The watermark sample is embedded into the key area of ​​the image sample; specifically, the position of each element in the binary matrix is ​​aligned with the position of the pixel point in the image sample, and the element value is multiplied by the pixel value to obtain a new pixel value to achieve watermark embedding.

5. A model watermark embedding method based on forgetting warning according to claim 1 or 4, characterized in that: The expression of the gradient weighted class activation mapping technique is as follows: In the formula, A represents a feature layer. In this example, it refers to the feature layer output by the last convolutional layer; k represents the kth channel in feature layer A; c represents category c; A k Represents the data of channel k in feature layer A; Represents A k The weight of The calculation formula is as follows: In the formula, y c Represents the score predicted by the convolutional neural network for category c without softmax activation; Represents the data of feature layer A at the coordinate (i, j) in channel k; Z is equal to the product of the width and height of the feature layer.

6. A model watermark embedding method based on forgetting warning according to claim 1 or 4, characterized in that: The process of calculating the target category corresponding to the watermark image sample includes: The forgetting index of each category is calculated according to the forgetting warning index, and the category with the highest forgetting index is selected as the target category corresponding to the watermark image sample. The watermark image sample is placed in the target category to obtain the watermark image data set. Among them, the calculation formula of the forgetting warning indicator is as follows: Where T is the number of categories, α base,i is the test accuracy of the first classification task (base set) after learning the i-th new classification task, α ideal is the accuracy of the offline classification model on the base set.

7. The model watermark embedding method based on forgetting warning according to claim 1 is characterized in that: The method further comprises: extracting and verifying watermarks from the trained classification model; Specifically, it includes: Input the watermark image dataset into the trained classification model to obtain the output category corresponding to each watermark image sample; According to the output category corresponding to each watermark image sample, the watermark image samples that do not match the true label are eliminated; Calculate the classification confidence distribution of the filtered watermark image samples after inputting them into the trained classification model, and select the watermark image sample with the highest classification confidence; The two-dimensional matrix in the watermark image sample is extracted and inversely transformed to obtain text information, thereby completing the watermark extraction and verification, and realizing the property rights protection of the model.

8. An electronic device comprising a memory and a processor, characterized in that: The memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the model watermark embedding method based on forgetting warning as described in any one of claims 1-7 above.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the model watermark embedding method based on forgetting warning as described in any one of claims 1 to 7 is implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the model watermark embedding method based on forgetting warning described in any one of claims 1-7 is implemented.

Images