A model watermark embedding method based on forgetting early warning, electronic equipment and medium

Abstract

The application discloses a model watermark embedding method based on forgetting early warning, an electronic device and a medium, and the method comprises the following steps: selecting an original image dataset; generating a watermark sample to be embedded; randomly selecting part of image samples from the original image dataset, determining a watermark position through an attention heat map technology, embedding the watermark sample into the selected image sample, and calculating a target category corresponding to the watermark image sample to obtain a watermark image dataset; alternately training a classification model by using the original image dataset and the watermark image dataset; and the trained classification model is used for realizing image classification and recognition and can realize watermark tracing.

Description

Technical Field

[0001] The present invention belongs to the technical field of model property protection, and in particular relates to a model watermark embedding method, electronic equipment, and medium based on forgetting warning. Background Art

[0002] With the rapid development of computer technology, artificial intelligence (AI) technologies such as deep learning have achieved tremendous success in areas such as computer vision, pattern recognition, and natural language processing. This has prompted tech giants such as Google and Microsoft to incorporate these technologies into their commercial products, improving service quality and economic benefits. However, building excellent AI models requires vast amounts of data, computing resources, and expertise. Furthermore, AI models are vulnerable to theft, tampering, and trafficking by unauthorized users. As AI rapidly advances, protecting the intellectual property rights of these models has become a major concern for both academia and industry.

[0003] The most common intelligent model intellectual property protection technology is backdoor-based watermarking. This is a covert digital watermarking technique that exploits the backdoor mechanism of an AI model to embed a watermark for model tracking and authentication. In this technique, the watermark is embedded in the model's weight parameters or input data, but is unrelated to the model's primary task and therefore has no impact on its performance. When the backdoor is activated, the watermark information can be activated and extracted to prove ownership of the model or track it. However, most of the existing backdoor-based watermark embedding technologies directly embed randomly generated perturbations into images, which lack semantic information and have insignificant features. They are difficult to extract and verify, and directly embedding semantic-free labels or text labels will affect the performance of the model's main task to a certain extent. In addition, the existing backdoor-based watermark embedding technologies usually randomly select a category or create a new category as the category label, which will cause the embedded watermark to be easily removed in subsequent work and is unstable. In addition, the existing watermark embedding technology is not robust enough against common attack methods such as pruning and quantization. Finally, the existing model watermark de-embedding technology is usually unable to extract the initial watermark during the verification process, but is determined by specific activation features or model outputs. This may have errors to a certain extent and cannot truly guarantee the property rights security of the model. Summary of the Invention

[0004] In order to solve the problems in existing backdoor-based watermark embedding technology, such as the lack of semantic information in the watermark, which affects the performance of the model's main task and is easily eliminated by means of pruning, quantization, etc., and the random selection of target classes or the creation of new classes, which leads to insufficient stability of the watermark information in subsequent model development, the present invention provides a model watermark embedding method, electronic device, and medium based on forgetting warning.

[0005] In a first aspect, an embodiment of the present invention provides a model watermark embedding method based on forgetting warning, the method comprising the following steps:

[0006] Select the original image dataset;

[0007] Generate watermark samples to be embedded;

[0008] Randomly select some image samples from the original image dataset, determine the watermark position through the attention heat map technology, embed the watermark sample into the selected image sample, and calculate the target category corresponding to the watermark image sample to obtain the watermark image dataset;

[0009] The classification model is trained alternately using the original image dataset and the watermarked image dataset; the trained classification model is used to realize image classification and recognition, and can also realize watermark traceability.

[0010] Furthermore, the process of generating the watermark sample to be embedded includes:

[0011] Get the watermark information to be embedded;

[0012] The watermark information is encoded using ASCII to obtain a two-dimensional matrix as the watermark sample to be embedded.

[0013] Furthermore, the watermark information includes but is not limited to date and unit name, and is used to characterize the classification model.

[0014] Furthermore, some image samples are randomly selected from the original image dataset, and the watermark position is determined by the attention heat map technology. The process of embedding the watermark sample into the selected image sample includes:

[0015] Input the image sample into the convolutional neural network, and output the attention heat map corresponding to each image sample through the gradient weighted class activation mapping technology. The attention heat map is used to illustrate the key areas in the image sample that affect the prediction of the convolutional neural network;

[0016] Embed the watermark sample into the key area of ​​the image sample; specifically, align the position of each element in the binary matrix with the position of the pixel in the image sample, and multiply the element value with the pixel value to obtain a new pixel value to achieve watermark embedding;

[0017] Furthermore, the expression of the gradient weighted class activation mapping technique is as follows:

[0018]

[0019] Where A represents a feature layer, in this case, the feature layer output by the last convolutional layer; k represents the kth channel in feature layer A; c represents category c; A k Represents the data of channel k in feature layer A; Representative for A k The weight of

[0020] The calculation formula is as follows:

[0021]

[0022] Where y c Represents the score predicted by the convolutional neural network for category c without softmax activation; Represents the data of feature layer A at the coordinate position (i, j) in channel k; Z is equal to the product of the width and height of the feature layer.

[0023] Furthermore, the process of calculating the target category corresponding to the watermark image sample includes:

[0024] Calculate the forgetting index of each category based on the forgetting warning indicator, select the category with the highest forgetting index as the target category corresponding to the watermark image sample, put the watermark image sample into the target category, and obtain the watermark image dataset;

[0025] The calculation formula of the forgetting warning indicator is as follows:

[0026]

[0027] Where T is the number of categories, α base,i is the test accuracy of the first classification task (base set) after learning the i-th new classification task, α ideal is the accuracy of the offline classification model on the base set.

[0028] Furthermore, the method further comprises: extracting and verifying watermarks from the trained classification model;

[0029] Specifically, it includes:

[0030] Input the watermark image dataset into the trained classification model to obtain the output category corresponding to each watermark image sample;

[0031] According to the output category corresponding to each watermark image sample, the watermark image samples that do not match the true label are eliminated;

[0032] Calculate the classification confidence distribution of the filtered watermark image samples after inputting them into the trained classification model, and select the watermark image sample with the highest classification confidence;

[0033] The two-dimensional matrix in the watermark image sample is extracted and inversely transformed to obtain text information, thereby completing the watermark extraction and verification, and realizing the intellectual property protection of the model.

[0034] In a second aspect, an embodiment of the present invention provides an electronic device comprising a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the above-mentioned model watermark embedding method based on forgetting warning.

[0035] In a third aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which implements the above-mentioned model watermark embedding method based on forgetting warning when the program is executed by a processor.

[0036] In a fourth aspect, an embodiment of the present invention provides a computer program product, including a computer program / instruction, which implements the above-mentioned model watermark embedding method based on forgetting warning when executed by a processor.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] This invention provides a model watermark embedding method based on forgetfulness warning. The method converts the text label information representing the model into a binary data conversion matrix, which is then embedded into the image as the watermark information. A forgetfulness warning method is then used to determine the current model's forgetfulness tendency for all categories of data. The least likely category to be forgotten is selected as the target label. The watermark embedding position is determined using an attention heat map, covering the model's original focus area to ensure the performance of the model's main task. A watermarked image dataset with embedded watermarks is obtained. A classification model is then trained alternately using the original image dataset and the watermarked image dataset. The trained classification model is used to implement image classification and recognition, and can also achieve watermark traceability. Finally, the watermark information embedded in the model is extracted and recovered, and its degree of recovery is evaluated to verify the model's ownership. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0040] Figure 1 A flow chart of a model watermark embedding method based on forgetting warning provided by an embodiment of the present invention;

[0041] Figure 2 A schematic diagram of generating a binary matrix for watermark text information provided by an embodiment of the present invention;

[0042] Figure 3 A schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0043] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0044] It should be noted that, unless there is any conflict, the features in the following embodiments and implementations may be combined with each other.

[0045] like Figure 1 and Figure 2 As shown, an embodiment of the present invention provides a model watermark embedding method based on forgetting warning, the method comprising:

[0046] Step S1: Select an original image dataset.

[0047] Exemplarily, in this example, the original image dataset CIFAR10 dataset is selected; the CIFAR10 dataset has a total of 60,000 samples, each sample is a 32*32 pixel RGB image (color image) of 10 categories of objects, and the label values ​​are distinguished by 0 to 9, which are airplane, automobile, bird, cat, deer, dog, frog, horse, ship and truck.

[0048] Furthermore, the original image dataset is parsed and preprocessed to map pixel images to their attribute labels. The dataset is then divided into a training set of 50,000 images and a test set of 10,000 images.

[0049] Step S2, generate the watermark samples to be embedded; randomly select some image samples from the original image dataset, determine the watermark position through the attention heat map technology, embed the watermark samples into the selected image samples, and calculate the target category corresponding to the watermark image samples to obtain the watermark image dataset.

[0050] Specifically, step S2 includes the following sub-steps:

[0051] Step S201: Generate a watermark sample to be embedded, including:

[0052] Acquire watermark information to be embedded, where the watermark information includes but is not limited to date and unit name, and is used to characterize the model.

[0053] The watermark information is encoded using ASCII (American Standard Code for Information Interchange) to obtain a two-dimensional matrix as a watermark sample to be embedded.

[0054] For example, in this example, "2024ZJUT" is used as the watermark information to be embedded. The number "2" in "2024ZJUT" can be expressed as "00000010", the character "Z" can be expressed as "01011010", the number "0" can be expressed as "00000000", the number "4" can be expressed as "00000100", the character "J" can be expressed as "01001010", the character "U" can be expressed as "01010101", and the character "T" can be expressed as "01010100". The final watermark information is converted into an 8×8 binary matrix as a watermark sample to be embedded.

[0055] Step S202: randomly select some image samples from the original image data set, determine the watermark position by using the gradient weighted class activation mapping technology, and embed the watermark sample into the selected image sample;

[0056] Specifically, the image samples are input into a convolutional neural network (CNN), and the gradient-weighted class activation mapping (Grad-CAM) technique is used to output an attention heat map corresponding to each image sample. The attention heat map is used to illustrate the key areas in the image sample that affect the convolutional neural network prediction, and the watermark sample is embedded in the key areas of the image sample. Specifically, the position of each element in the binarized matrix is ​​aligned with the position of the pixel point in the image sample, and the element value is multiplied by the pixel value to obtain the new pixel value to achieve watermark embedding.

[0057] Furthermore, the Gradient Weighted Class Activation Mapping (Grad-CAM) technique works as follows:

[0058] Forward propagation: The input image is passed through the CNN to generate feature maps at various convolutional layers.

[0059] Compute gradients: Calculate the gradient of the target class (the class being predicted) with respect to the feature map of the last convolutional layer.

[0060] Global Average Pooling (GAP): The gradients are then globally averaged to provide an importance score for each feature map. This step captures the importance of each feature map to the target class.

[0061] Weighted sum of activation maps: The importance scores obtained in the previous step are used as weights to compute the weighted sum of activation maps of the last convolutional layer.

[0062] ReLU and upsampling: The resulting map is passed through a ReLU activation function to remove any negative values ​​and then upsampled to the size of the input image.

[0063] Visualization: Overlay the upsampled heatmap onto the input image to visualize the areas in the image that are important for the CNN prediction.

[0064] Furthermore, the expression of Grad-CAM is as follows:

[0065]

[0066] Where A represents a feature layer, in this case, the feature layer output by the last convolutional layer; k represents the kth channel in feature layer A; c represents category c; A k Represents the data of channel k in feature layer A; Representative for A k The weight of .

[0067] The calculation formula is as follows:

[0068]

[0069] Where y c Represents the score predicted by the convolutional neural network for category c without softmax activation; Represents the data of feature layer A at the coordinate position (i, j) in channel k; Z is equal to the product of the width and height of the feature layer.

[0070] Step S203, calculating the target category corresponding to the watermark image sample;

[0071] Specifically, the forgetting index of each category is calculated according to the forgetting warning indicator, and the category with the highest forgetting index is selected as the target category corresponding to the watermark image sample. The watermark image sample is placed in the target category to obtain the watermark image dataset.

[0072] Specifically, the calculation formula of the forgetting warning indicator is as follows:

[0073]

[0074] Where T is the number of categories, α base,i is the test accuracy of the first classification task (base set) after learning the i-th new classification task, α ideal is the accuracy of the offline classification model on the base set.

[0075] Step S3, alternately training a classification model using the original image dataset and the watermark image dataset; the trained classification model is used to implement image classification and recognition, and can also achieve watermark traceability.

[0076] Specifically, in this example, a three-layer deep neural network framework is built with 64 input nodes, 10 output nodes, and 32 hidden nodes in the middle layer. The ReLU activation function is used to increase the nonlinearity of the neural network model. Its expression is as follows:

[0077] f(x)=max(0,x)

[0078] Dropout with a probability of 0.8 is set after the output of each layer. During each generation of training, there is a probability of 0.2 to actively and temporarily ignore some hidden nodes, which speeds up the training of the classification model and prevents overfitting.

[0079] The final result is output by the softmax function, and its expression is as follows:

[0080]

[0081] Among them, z i is the output value of the i-th node, and C is the number of output probabilities, that is, the number of classification categories, which is 10 in this example. The Softmax function is used to convert the output values ​​of the 10 classifications into a probability distribution in the range [0, 1] and with a value of 1 to obtain the prediction result.

[0082] The loss value between the predicted value obtained by forward propagation and the true target label is then calculated based on the cross entropy loss function, which is expressed as follows:

[0083]

[0084] Among them, M represents the number of categories, y ic is a sign function (0 or 1), which takes 1 if the true category of sample i is equal to c, otherwise it takes 0, p ic represents the probability that the observed sample i belongs to category c. Next, in the back-propagation process, the partial derivative of the loss function L with respect to the model parameters is calculated to obtain a gradient value. Then, the model parameters are updated using the momentum-based stochastic gradient descent method SGD with a learning rate of 0.1 and a momentum factor of 0.5. The expression is as follows:

[0085] v i =αv i-1 +η▽L(θ)

[0086] θ i =θ i-1 -v i

[0087] Among them, α represents the momentum parameter, η is the learning rate, ▽L(θ) is the gradient value obtained by taking the partial derivative of the loss function with respect to the parameter θ, and θ i-1 and θ i Represent the model parameters before and after the update respectively.

[0088] Step S4: extract and verify the watermark of the trained classification model.

[0089] Specifically, the watermark image dataset is input into the trained classification model to obtain the output category corresponding to each watermark image sample;

[0090] According to the output category corresponding to each watermark image sample, the watermark image samples that do not match the true label are eliminated;

[0091] Calculate the classification confidence distribution of the filtered watermark image samples after inputting them into the trained classification model, and select the watermark image sample with the highest classification confidence;

[0092] The two-dimensional matrix in the watermark image sample is extracted and inversely transformed to obtain text information, thereby completing the watermark extraction and verification, and realizing the intellectual property protection of the model.

[0093] In summary, this paper provides a model watermark embedding method based on forgetfulness warning. The method converts the textual label information representing the model into a binary data conversion matrix, which is then embedded as the watermark information in the image. A forgetfulness warning method is then used to determine the current model's forgetfulness tendency for all data categories. The least likely category to be forgotten is selected as the target label, resulting in a watermarked image dataset embedded with the watermark. A classification model is then trained alternately using the original image dataset and the watermarked image dataset. The trained classification model is used to implement image classification and recognition, and can also trace the watermark. Finally, the watermark information embedded in the model is extracted and recovered, and its degree of recovery is evaluated to verify the model's ownership.

[0094] Accordingly, the present application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned model watermark embedding method based on forgetting warning. Figure 3 As shown in FIG, a hardware structure diagram of any device with data processing capability in which the model watermark embedding method based on forgetting warning provided by the embodiment of the present invention is located, except Figure 3 In addition to the processor, memory, and network interface shown, any device with data processing capabilities in which the apparatus in the embodiment is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0095] Correspondingly, the present application also provides a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the model watermark embedding method based on forgetting warning as described above. The computer-readable storage medium may be an internal storage unit of any device with data processing capabilities as described in any of the aforementioned embodiments, such as a hard disk or memory. The computer-readable storage medium may also be an external storage device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), an SD card, a flash card (Flash Card), etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capabilities. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and may also be used to temporarily store data that has been output or is to be output.

[0096] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the contents disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only.

[0097] It will be understood that the present application is not limited to the exact construction that has been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof.

Claims

1. A model watermark embedding method based on forgetting warning, characterized in that: The method comprises the following steps: Select the original image dataset; Generate watermark samples to be embedded; Randomly select some image samples from the original image dataset, determine the watermark position through the attention heat map technology, embed the watermark sample into the selected image sample, and calculate the target category corresponding to the watermark image sample to obtain the watermark image dataset; The classification model is trained alternately using the original image dataset and the watermarked image dataset; the trained classification model is used to achieve image classification and recognition, and can also achieve watermark traceability; The process of randomly selecting some image samples from the original image dataset, determining the watermark position through the attention heat map technology, and embedding the watermark samples into the selected image samples includes: Input the image sample into the convolutional neural network, and output the attention heat map corresponding to each image sample through the gradient weighted class activation mapping technology. The attention heat map is used to illustrate the key areas in the image sample that affect the prediction of the convolutional neural network; Embed the watermark sample into the key area of ​​the image sample; specifically, align the position of each element in the binary matrix with the position of the pixel in the image sample, and multiply the element value with the pixel value to obtain a new pixel value to achieve watermark embedding; The process of calculating the target category corresponding to the watermark image sample includes: Calculate the forgetting index of each category based on the forgetting warning indicator, select the category with the highest forgetting index as the target category corresponding to the watermark image sample, put the watermark image sample into the target category, and obtain the watermark image dataset; The calculation formula of the forgetting warning indicator is as follows: ； in, is the number of categories, It is The test accuracy of the first classification task, i.e. the base set, after learning a new classification task. is the accuracy of the offline classification model on the base set.

2. The model watermark embedding method based on forgetting warning according to claim 1 is characterized in that: The process of generating the watermark sample to be embedded includes: Get the watermark information to be embedded; The watermark information is encoded using ASCII to obtain a two-dimensional matrix as the watermark sample to be embedded.

3. The model watermark embedding method based on forgetting warning according to claim 2 is characterized in that: The watermark information includes but is not limited to date and unit name, and is used to represent the classification model.

4. The model watermark embedding method based on forgetting warning according to claim 1 is characterized in that: The expression of the gradient weighted class activation mapping technique is as follows: ； Where, The feature layer representing the output of the last convolutional layer; Representative feature layer The kth channel in ; Representative categories ; Representative feature layer Middle Channel data; Representatives for The weight of The calculation formula is as follows: ； Where, Representative convolutional neural network for categories The predicted scores without softmax activation; Representative feature layer In the channel In the example, the coordinates are the data at the position (i, j); Equal to the product of the width and height of the feature layer.

5. The model watermark embedding method based on forgetting warning according to claim 1 is characterized in that: The method further comprises: extracting and verifying watermarks from the trained classification model; Specifically, it includes: Input the watermark image dataset into the trained classification model to obtain the output category corresponding to each watermark image sample; According to the output category corresponding to each watermark image sample, the watermark image samples that do not match the true label are eliminated; Calculate the classification confidence distribution of the filtered watermark image samples after inputting them into the trained classification model, and select the watermark image sample with the highest classification confidence; The two-dimensional matrix in the watermark image sample is extracted and inversely transformed to obtain text information, thereby completing the watermark extraction and verification, and realizing the intellectual property protection of the model.

6. An electronic device comprising a memory and a processor, characterized in that: The memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the model watermark embedding method based on forgetting warning as described in any one of claims 1 to 5.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the model watermark embedding method based on forgetting warning as described in any one of claims 1 to 5 is implemented.

8. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the model watermark embedding method based on forgetting warning described in any one of claims 1 to 5 is implemented.

Images