A model copyright protection method and electronic device based on neuron screening

Abstract

This application provides a model copyright protection method and electronic device based on neuron selection, relating to the field of digital copyright technology. The method includes: dividing a CNN-based image classification model into multiple neuron layers, with weight coefficients for the parameters to be solved in each neuron layer; training the image classification model using a first objective loss function with a first training dataset containing image samples, adjusting the weight coefficients and obtaining a set of activated neurons during training; obtaining a second training dataset; and training an image classification model using a second objective loss function with fixed neuron parameter values ​​in the set of activated neurons using the second training dataset to obtain a trained image classification model. The implementation provided in this application reduces the interference and impact of watermark recognition on normal image classification, enhancing the concealment of the watermark recognition function.

Description

Technical Field

[0001] This application relates to the fields of digital rights management technology, deep learning technology, and image processing technology, specifically to a model copyright protection method based on neuron screening, an electronic device, and a corresponding storage medium. Background Technology

[0002] Currently, with the rapid development of artificial intelligence (AI) technology, its applications are becoming increasingly widespread, including image analysis, speech recognition, text processing, intelligent recommendation, and security detection. Federated learning and large-scale modeling have become cutting-edge hot topics in AI. Copyright protection for AI systems is a key issue that needs to be addressed in the commercial application of AI technology. Protecting intellectual property rights for AI technology, maintaining order in the AI ​​market, and promoting the dissemination and application of AI technology are of great significance. However, current mainstream AI copyright protection technology directly trains the final model by using watermarked samples along with the original samples as the training sample set. In most cases, watermarked samples interfere with the model's normal image classification function, even significantly reducing its functionality and performance.

[0003] Existing technologies also include solutions that use any global feature as the basis for watermarking. However, these solutions require manual design of the global feature quantization function, which can be quite difficult in scenarios with many image categories and model parameters. If the global feature quantization function is not designed appropriately, it will significantly affect the watermarking effect. Summary of the Invention

[0004] The purpose of this application is to provide a model copyright protection method and electronic device based on neuron screening. It comprehensively adopts techniques such as hierarchical sparsity, neuron activation value screening, and model local parameter training to separate the training process of the model's normal image classification function from the training process of the watermark recognition function, so as to reduce the interference and influence of the watermark recognition function on the normal image classification function, enhance the concealment of the watermark recognition function, and at least solve some of the problems in the background technology.

[0005] To achieve the above objectives, this application provides a model copyright protection method based on neuron selection. The method includes: dividing a CNN-based image classification model into multiple neuron layers, where each neuron layer has weight coefficients for the parameters to be solved; constructing a first target loss function based on classification loss, the magnitude of the parameters to be solved, and the weight coefficients; training the image classification model using the first target loss function on a first training dataset containing image samples, adjusting the weight coefficients and obtaining a set of activated neurons during training; obtaining a second training dataset, which includes image sample pairs consisting of image samples and new images obtained by adding a watermark image to the image samples, where the category label of the new images is a preset fixed category; and constructing a second target loss function based on classification loss; training the image classification model using the second training dataset on the second target loss function and with fixed neuron parameter values ​​in the set of activated neurons to obtain a trained image classification model; and verifying the trained image classification model by classifying an input image containing the watermark image into the preset fixed category.

[0006] Preferably, the trained image classification model is validated by classifying an input image containing the watermark image into the preset fixed category, including: acquiring multiple test images that do not belong to the preset fixed category and do not contain the watermark image to form a first validation set; adding a watermark image to each test image in the first validation set to obtain a second validation set; inputting the second validation set into the image classification model to be validated to obtain a classification result corresponding to each input image in the second validation set; calculating the proportion of classification results classified into the preset fixed category among all classification results; when the proportion is higher than a preset ratio, it indicates that the image classification model to be validated is related to the watermark image used by the trained image classification model.

[0007] Preferably, the CNN-based image classification model is divided into multiple neuron layers, including: dividing the CNN-based image classification model into non-fully connected layers and fully connected layers, wherein the fully connected layers are located in the last few layers of the image classification model; and dividing the non-fully connected layers into multiple neuron layers.

[0008] Preferably, constructing a first target loss function based on classification loss, the magnitude of the parameters to be solved, and the weight coefficients includes: using the magnitude of the parameters to be solved and the weight coefficients as the accumulated value of the parameters as a first part; using the cross-entropy loss function as a second part; and summing the first part and the second part to obtain the first target loss function.

[0009] Preferably, the first part is based on the sum of the magnitude of the parameter to be solved and the weight coefficients, and includes: using the second norm of the parameter to be solved as the magnitude of the parameter to be solved; obtaining the cumulative value of the current layer by weighted summation of the squares of the magnitudes of the parameters to be solved in the same neuron layer and the weight coefficients of the neuron layer; and obtaining the first part by summing the cumulative values ​​of the current layers of all neuron layers.

[0010] Preferably, adjusting the weight coefficients during training includes: when the vector composed of all parameters to be solved in the image classification model is updated during training, obtaining the proportion of the number of parameters to be solved in each neuron layer whose values ​​are within a preset range to the total number of all parameters to be solved in that layer; when the proportion is less than the lower limit of a preset proportion threshold, increasing the weight coefficients corresponding to that neuron layer; when the proportion is greater than the upper limit of a preset proportion threshold, decreasing the weight coefficients corresponding to that neuron layer.

[0011] Preferably, obtaining the set of activated neurons during training includes: adding neurons in the image classification model whose output value is not 0 and which are not in the fully connected layer to the set to obtain the set of activated neurons.

[0012] Preferably, the image samples in the image sample pairs in the second training dataset are selected from the first training dataset.

[0013] Preferably, constructing a second objective loss function based on classification loss includes: obtaining a first part by calculating the mean of the negative logarithms of the classification probabilities of new images in an image sample pair using an image classification model; obtaining a second part by calculating the cross-entropy loss function of the classification results of the image samples in the image sample pair using an image classification model; and constructing the second objective loss function based on the first part and the second part.

[0014] Preferably, when training an image classification model using the first objective loss function with a first training dataset containing image samples, and when training an image classification model using the second objective loss function with a second training dataset and whose neuron parameter values ​​in the set of activated neurons are fixed, the Mini-batch SGD algorithm is used for iterative iteration.

[0015] This application also provides a model copyright protection device based on neuron selection. The device includes: a training preparation module for dividing a CNN-based image classification model into multiple neuron layers, where each neuron layer has weight coefficients for the parameters to be solved, and constructing a first target loss function based on classification loss, the magnitude of the parameters to be solved, and the weight coefficients; a first training module for training the image classification model using the first target loss function on a first training dataset containing image samples, adjusting the weight coefficients and obtaining a set of activated neurons during training; a watermark preparation module for obtaining a second training dataset, which includes image sample pairs consisting of image samples and new images obtained by adding a watermark image to the image samples, where the category label of the new images is a preset fixed category, and constructing a second target loss function based on classification loss; and a second training module for training the image classification model using the second training dataset and employing the second target loss function with fixed neuron parameter values ​​in the set of activated neurons, to obtain a trained image classification model; the trained image classification model is verified by classifying an input image containing the watermark image into the preset fixed category.

[0016] This application also provides an electronic device, including: at least one processor; and a memory connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the at least one processor implements the aforementioned neuron-based model copyright protection method by executing the instructions stored in the memory.

[0017] This application also provides a machine-readable storage medium storing instructions that, when executed by a processor, configure the processor to perform the aforementioned neuron-based model copyright protection method.

[0018] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the aforementioned neuron-based model copyright protection method.

[0019] The above technical solution has the following beneficial effects:

[0020] (1) This application separates the training process of the model for normal image classification tasks from the training process for watermark recognition functions. During the training process for normal image classification tasks, the sparsity of parameters of each layer of the model is periodically quantitatively evaluated and controlled, which can ensure that each layer of the model can reserve enough redundant neurons to ensure that the model training for watermark recognition functions can be completed smoothly.

[0021] (2) This application first trains the model for the normal image classification task, then fixes the neuron parameter values ​​related to the normal image classification task, and then trains other unfixed parameters of the model for the watermark recognition task, which can reduce the negative impact of the added watermark recognition function on the normal image classification function of the model.

[0022] (3) The process of adding watermark recognition function to the model provided in this application does not change the internal structure and number of parameters of the model. It is difficult to detect from the external behavior and structure of the model that the watermark recognition function has been added to the model, which greatly increases the concealment of the model watermark recognition function.

[0023] Other features and advantages of the embodiments of this application will be described in detail in the following detailed description section. Attached Figure Description

[0024] The accompanying drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the following detailed description to explain the embodiments of this application, but do not constitute a limitation on the embodiments of this application. In the drawings:

[0025] Figure 1 This illustration schematically shows the steps of a model copyright protection method based on neuron screening according to an embodiment of this application;

[0026] Figure 2 The illustration shows a schematic diagram of the implementation process of the model copyright protection method based on neuron screening according to the embodiments of this application;

[0027] Figure 3 This illustration schematically shows a structural diagram of a model copyright protection device based on neuron-based screening according to an embodiment of this application;

[0028] Figure 4 The diagram schematically illustrates the internal structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0029] The specific embodiments of this application will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit the embodiments of this application.

[0030] Figure 1 The illustration shows a schematic diagram of the steps of a model copyright protection method based on neuron-based screening according to an embodiment of this application. For example... Figure 1 As shown, a model copyright protection method based on neuron screening is proposed, which includes:

[0031] S01. Divide the CNN-based image classification model into multiple neuron layers. The parameters to be solved in each neuron layer have weight coefficients. Construct a first target loss function based on the classification loss, the size of the parameters to be solved, and the weight coefficients.

[0032] S02. The image classification model using the first objective loss function is trained using a first training dataset containing image samples, and the weight coefficients are adjusted and the set of activated neurons is obtained during the training process.

[0033] S03. Obtain a second training dataset, which includes image sample pairs consisting of image samples and new images obtained by adding watermark images to the image samples. The category label of the new images is a preset fixed category. A second objective loss function is constructed based on classification loss.

[0034] S04. The image classification model with the second objective loss function and fixed neuron parameter values ​​in the set of activated neurons is trained using the second training dataset to obtain the trained image classification model; the trained image classification model is verified by classifying an input image containing the watermark image into the preset fixed category.

[0035] In step S01, the weight coefficients can be set at the neuron layer level, meaning that the weight coefficients of the parameters to be solved in the same neuron layer are the same. These weight coefficients are periodically monitored and adjusted during the training process in step S02, thereby reducing the risk of watermark recognition function training failure caused by excessive differences in parameter sparsity distribution between different layers of the model.

[0036] The watermark image in step S03 is used to provide a specific image identifier, which can be one or more of the following: company logo, company Chinese name, company English name, specific shape image, specific string, etc.

[0037] In step S04, all parameters to be solved in the image classification model are divided into two categories based on the set of activated neurons: parameters with fixed values ​​and parameters with unfixed values. During the training process in step S04, only the parameters with unfixed values ​​can be changed, thereby avoiding negative impacts on the parameters of neurons within the set of activated neurons.

[0038] Those skilled in the art will also understand that the preparation and acquisition of the second training dataset in step S03 does not use the parameters from the aforementioned steps, and therefore does not have a strict execution position; it can be performed before step S04. This is only for the sake of textual description. For example, steps S02 and S03 can be performed simultaneously, or step S03 can be performed before step S02, that is, the second training dataset and the second objective loss function can be prepared first. Step S03 can also be prepared and acquired simultaneously with the first training dataset, or prepared and acquired before or after acquiring the first training dataset.

[0039] The CNN-based image classification models in this and subsequent embodiments include, but are not limited to, AlexNet, VGG, GoogLeNet, ResNet, FPN, DenseNet, etc.

[0040] Through the above implementation methods, the internal structure and number of parameters of the model can be kept unchanged, and it is difficult to detect from the external behavior and structure of the model that the watermark recognition function has been added, which greatly increases the concealment of the watermark recognition function.

[0041] In some optional embodiments of this application, the trained image classification model is verified by classifying an input image containing the watermark image into the preset fixed category. This includes: acquiring multiple test images that do not belong to the preset fixed category and do not contain the watermark image, forming a first verification set; adding a watermark image to each test image in the first verification set to obtain a second verification set; inputting the second verification set into the image classification model to be verified to obtain a classification result corresponding to each input image in the second verification set; calculating the proportion of classification results classified into the preset fixed category among all classification results; and indicating that the proportion is higher than a preset ratio, the image classification model to be verified is associated with the watermark image used by the trained image classification model. The aforementioned association includes, but is not limited to: the image classification model to be verified being marked by the watermark image; the copyright of the image classification model to be verified belonging to the organization or individual providing the watermark image; and the owner of the image classification model to be verified being related to the information in the watermark image. Specifically, for any copyright model M′ to be verified, prepare multiple images that do not belong to the Kth category and have not been watermarked. These images constitute the image set V, i.e., the first verification set. Add watermarked images X to the image samples in the first verification set. w After synthesizing the new image, the first validation set becomes the second validation set V'. The proportion of all images in the second validation set predicted as the Kth class by model M' is calculated. P is then used as the lower limit for the proportion of all images in V' identified as the Kth class. A value below P indicates no watermark is embedded; a value above P indicates that model M' has added a watermark X.w The copyright of model M′ belongs to the organization or individual that provides the watermark. Here, P∈(0,1], and P represents the confidence level of the model being embedded with the watermark. The Kth category is the preset fixed category.

[0042] In some optional embodiments of this application, the CNN-based image classification model is divided into multiple neuron layers, including: dividing the CNN-based image classification model into non-fully connected layers and fully connected layers, wherein the fully connected layers are located in the last few layers of the image classification model; and dividing the non-fully connected layers into multiple neuron layers. Assuming M is a CNN-based image classification model, all hidden layer neurons in model M use the ReLU activation function, and the last few layers of model M are fully connected layers, model M is divided into L neuron layers, and all the parameters to be solved in the l-th layer of model M constitute a set Θ. l Θ l The m-th parameter to be solved is denoted as set Θ l The cardinality is denoted as |Θ l This implementation divides the model into two parts: a non-fully connected layer and a fully connected layer. Different training strategies are used for the parameters of the two parts. While reducing the negative interference of watermark image features on normal image features, the classification accuracy of the fully connected layer is maximized.

[0043] In some optional embodiments of this application, a first target loss function is constructed based on classification loss, the magnitude of the parameter to be solved, and the weight coefficients, including: using the magnitude of the parameter to be solved and the weight coefficients as the accumulated value of the parameter as a first part; using the cross-entropy loss function as a second part; and summing the first part and the second part to obtain the first target loss function. Furthermore, the second norm of the parameter to be solved is used as the magnitude of the parameter to be solved; the square of the magnitude of the parameter to be solved in the same neuron layer and the weight coefficient of that neuron layer are summed by weighted summation to obtain the cumulative value of that layer; and the first part is obtained by summing the cumulative values ​​of all neuron layers. D is the first training sample image set of model M, and the first target loss function L is used in the training process of model M. D (θ) is shown in the following equation. θ is a vector consisting of all the parameters to be solved in model M, 1≤l≤L, 1≤m≤|Θ l |, where l, L, and m are positive integers.

[0044]

[0045] Where 1(x) is an indicator function, specifically:

[0046]

[0047] Where B is the number of sample images in each batch, C is the total number of sample image categories contained in the training set D and the test set T, and y i Let be the class label index of the i-th sample image in a batch of training set D. Let α be the probability that model M predicts the i-th sample image as belonging to the k-th category. l These are the weight coefficients of the parameters to be solved in the l-th layer of model M. It is the square of the second norm of vector x. B, C, y i , i, and k are all positive integers, 1 ≤ i ≤ B, 1 ≤ k ≤ C, 1 ≤ y i ≤C, α l >0. The first objective loss function mainly introduces the magnitude of the parameters to be solved and the weight coefficients as parameters, through... Loss calculation based on total size The values ​​of are used for overall suppression.

[0048] In some embodiments of this application, adjusting the weight coefficients during training includes: when the vector composed of all unsolved parameters in the image classification model is updated during training, obtaining the proportion of the number of unsolved parameters with values ​​within a preset range in each neuron layer to the total number of unsolved parameters in that layer; when the proportion is less than a preset lower threshold, increasing the weight coefficient corresponding to that neuron layer; when the proportion is greater than a preset upper threshold, decreasing the weight coefficient corresponding to that neuron layer. Specifically, after the r-th iteration training cycle of model M (each update of the parameter θ of model M indicates the end of an iteration training cycle), the proportion of the number of unsolved parameters with non-zero values ​​in each neuron layer of model M to the total number of unsolved parameters in that layer is calculated, denoted by S. l This represents the proportion of non-zero parameters in the l-th layer of model M to the total number of parameters in that layer. If S l MIN Then increase α l The value of S, if l >S MAX Then decrease α l The value of is determined by the model training process. If the model training process does not meet the termination condition (the model parameters converge or r exceeds the preset threshold), then r = r + 1, and the next iteration training cycle is started. Where S... MIN ∈[0,1]、S MAX ∈[0,1]、S MAX ≥S MIN ​This implementation method periodically and quantitatively evaluates and controls the parameter sparsity of each layer of the model during the training process for normal image classification tasks. This ensures that each layer of the model can reserve enough redundant neurons to guarantee the successful completion of model training for watermark recognition.

[0049] In some embodiments of this application, obtaining the set of activated neurons during training includes: adding neurons in the image classification model whose neuron output value is not 0 and which are not in the fully connected layer to a set, thereby obtaining the set of activated neurons. Specifically, for each image sample X in the dataset D, X is input into the model M, and the neurons in the model M whose neuron output value is not 0 and which are not in the fully connected layer are recorded. These neurons are then added to set A, which is the set of activated neurons.

[0050] In some embodiments of this application, image samples in the image sample pairs of the second training dataset are selected from the first training dataset. This embodiment defines the relationship between the second training dataset and the first training dataset. Assume X w The watermark image is smaller than the size of the training sample images in the training dataset D. The watermark image can contain one or more of the following: a company logo, the company's Chinese or English name, a specific shape image, a specific string, etc. For each image sample X in D, X will... w Add the samples to the image sample X to synthesize a new image X′. All image sample pairs (X, X′) constitute the image set D′, which is the second training dataset. Set the class label of all X′ in D′ to K∈[1,C], where K is a positive integer constant.

[0051] In some embodiments of this application, constructing a second objective loss function based on classification loss includes: obtaining a first part by calculating the mean of the logarithmic classification probabilities of new images in an image sample pair using an image classification model; obtaining a second part by calculating the cross-entropy loss function of the classification results of image samples in an image sample pair using an image classification model; and constructing the second objective loss function based on the first and second parts. The second objective loss function L used in training model M is... D′ (θ) is shown in the following formula.

[0052]

[0053] Among them, y j Let X be the class label number of the j-th sample image pair in image set D′. It is the probability that model M predicts X′ in the j-th image sample pair in a batch of image set D′ as the k-th class. Let B be the probability that model M predicts X in the j-th image sample pair in a batch of image set D′ as belonging to the k-th class. 1≤j≤B, where j is a positive integer.

[0054] Using image set D′ as the training sample set, the parameter values ​​of neurons in set A are fixed. Each batch contains B image sample pairs, for a total of 2B image samples. The Mini-Batch SGD algorithm is used to iteratively train all the unfixed parameters of model M.

[0055] In some embodiments of this application, when training an image classification model using the first objective loss function with a first training dataset containing image samples, and when training an image classification model using the second objective loss function with a second training dataset and whose neuron parameter values ​​in the activated neuron set are fixed, the Mini-batch SGD algorithm is used iteratively. Compared with other learning algorithms, mini-batch SGD has the following advantages: (1) lower computational cost because only a small batch of gradients is calculated each time; (2) faster convergence to the global optimum because each batch may contain some samples far from the local optimum; (3) lower memory consumption for large-scale datasets because only a small batch of samples is loaded each time. This embodiment limits the way model parameters are updated, thus improving model training efficiency.

[0056] The model copyright protection method based on neuron screening described above can significantly increase the concealment of the model's watermark recognition function by making it difficult to detect from the model's external behavior and structure without changing the model's internal structure and number of parameters.

[0057] Figure 2 The illustration shows a schematic diagram of the implementation process of the model copyright protection method based on neuron screening according to an embodiment of this application. For example... Figure 2 As shown, a specific implementation process of the model copyright protection method based on neuron screening is as follows:

[0058] (1) Assume M is a CNN-based image classification model (such as AlexNet, VGG, GoogLeNet, ResNet, FPN, DenseNet, etc.). All hidden layer neurons in model M use the ReLU activation function, and the last few layers of model M are fully connected layers. Model M is divided into L neuron layers. D is the training sample image set of model M (for example, containing 128,000 images in 10 categories, with an image size of 224×224×3). Based on the training sample set D, the parameters of model M are trained iteratively using the Mini-batch SGD algorithm.

[0059] (2) After the r-th iteration training cycle of model M (each update of the parameter θ of model M indicates the end of an iteration training cycle), the proportion of the number of non-zero parameters in each neuron layer of model M to the total number of parameters to be solved in that layer is calculated using S. l This represents the proportion of non-zero parameters in the l-th layer of model M to the total number of parameters in that layer. If S l MIN Then increase α l The value of S, if l >S MAX Then decrease α l The value of r is determined by the training process. If the model training process does not meet the termination condition (the model parameters converge or r exceeds 200), then r = r + 1 is set, and the next iteration training cycle is started.

[0060] (3) For each image sample X in dataset D, input X into model M, record the neurons in model M whose output value is not 0 and are not in the fully connected layer, and add the above neurons to set A.

[0061] (4) Assume X w The watermark image is 32×8×3 pixels. The watermark image is a Topsec string. For each image sample X in D, X is... w Add the samples to the image sample X to synthesize a new image X′. All image sample pairs (X, X′) constitute the image set D′. Set the class label of all X′ in D′ to 3.

[0062] (5) Using image set D′ as the training sample set, the parameter values ​​of neurons in set A are fixed. Each batch contains B = 128 image sample pairs, for a total of 256 image samples. The Mini-Batch SGD algorithm is used to iteratively train all unfixed parameters of model M.

[0063] (6) For any copyright model M′ to be verified, prepare 100 images that do not belong to the third category and have not been watermarked. These images constitute the image set V. Add a watermarked image X to the image samples in set V. w After synthesizing the new image, the image set is now denoted as V'. If the proportion of the new images in V' that are predicted as category 3 by model M' is higher than the preset ratio P (here, P can be set to 90%), then it can be determined that model M' has added the watermark X. w The copyright of model M′ belongs to the organization or individual that provided the watermark.

[0064] ​As can be seen from the above implementation methods, this application has the advantage of not changing the internal structure and number of parameters of the model, and making it difficult to detect from the external behavior and structure of the model that the watermark recognition function has been added to the model, thus greatly increasing the concealment of the model watermark recognition function.

[0065] Based on the same inventive concept, this application also provides a model copyright protection device based on neuron screening. Figure 3 A schematic diagram illustrating the structure of a model copyright protection device based on neuron-based screening according to an embodiment of this application is shown. Figure 3 As shown, the device includes: a training preparation module, used to divide a CNN-based image classification model into multiple neuron layers, each neuron layer having weight coefficients for the parameters to be solved, and constructing a first target loss function based on classification loss, the size of the parameters to be solved, and the weight coefficients; a first training module, used to train the image classification model using the first target loss function with a first training dataset containing image samples, adjusting the weight coefficients and obtaining a set of activated neurons during training; a watermark preparation module, used to obtain a second training dataset, the second training dataset including image sample pairs consisting of image samples and new images obtained by adding a watermark image to the image samples, the category label of the new images being a preset fixed category, and constructing a second target loss function based on classification loss; and a second training module, used to train the image classification model using the second training dataset with the second target loss function and fixed values ​​for the neuron parameters in the set of activated neurons, to obtain a trained image classification model; the trained image classification model is verified by classifying an input image containing the watermark image into the preset fixed category.

[0066] In some optional embodiments of this application, the trained image classification model is verified by classifying an input image containing the watermark image into the preset fixed category. This includes: acquiring multiple test images that do not belong to the preset fixed category and do not contain the watermark image to form a first verification set; adding a watermark image to each test image in the first verification set to obtain a second verification set; inputting the second verification set into the image classification model to be verified to obtain a classification result corresponding to each input image in the second verification set; calculating the proportion of classification results classified into the preset fixed category among all classification results; and when the proportion is higher than a preset ratio, it indicates that the image classification model to be verified is associated with the watermark image used by the trained image classification model.

[0067] In some optional embodiments of this application, the CNN-based image classification model is divided into multiple neuron layers, including: dividing the CNN-based image classification model into non-fully connected layers and fully connected layers, wherein the fully connected layers are located in the last few layers of the image classification model; and dividing the non-fully connected layers into multiple neuron layers.

[0068] In some optional embodiments of this application, constructing a first target loss function based on classification loss, the magnitude of the parameters to be solved, and the weight coefficients includes: using the magnitude of the parameters to be solved and the weight coefficients as the accumulated value of the parameters as a first part; using the cross-entropy loss function as a second part; and summing the first part and the second part to obtain the first target loss function.

[0069] In some optional embodiments of this application, the first part is based on the magnitude of the parameter to be solved and the weight coefficient as the cumulative value of the parameter, including: using the second norm of the parameter to be solved as the magnitude of the parameter to be solved; obtaining the cumulative value of the current layer by weighted summation of the square of the magnitude of the parameter to be solved in the same neuron layer and the weight coefficient of the neuron layer; and obtaining the first part by summing the cumulative values ​​of the current layer of all neuron layers.

[0070] In some optional embodiments of this application, adjusting the weight coefficients during training includes: when the vector composed of all unsolved parameters in the image classification model is updated during training, obtaining the proportion of the number of unsolved parameters in each neuron layer whose values ​​are within a preset range to the total number of unsolved parameters in that layer; when the proportion is less than the lower limit of a preset proportion threshold, adjusting the weight coefficients corresponding to that neuron layer upwards; when the proportion is greater than the upper limit of a preset proportion threshold, adjusting the weight coefficients corresponding to that neuron layer downwards.

[0071] In some optional embodiments of this application, obtaining the set of activated neurons during training includes: adding neurons whose output values ​​are not 0 and are not in the fully connected layer of the image classification model during training to the set to obtain the set of activated neurons.

[0072] In some alternative embodiments of this application, the image samples in the image sample pairs in the second training dataset are selected from the first training dataset.

[0073] In some optional embodiments of this application, constructing a second objective loss function based on classification loss includes: obtaining a first part by calculating the mean of the logarithmic classification probabilities of new images in an image sample pair using an image classification model; obtaining a second part by calculating the cross-entropy loss function of the classification results of image samples in an image sample pair using an image classification model; and constructing the second objective loss function based on the first part and the second part.

[0074] In some optional embodiments of this application, when training an image classification model using the first objective loss function with a first training dataset containing image samples, and when training an image classification model using the second objective loss function with a second training dataset and whose neuron parameter values ​​in the set of activated neurons are fixed, the Mini-batch SGD algorithm is used for iterative iteration.

[0075] The specific limitations of each functional module in the aforementioned neuron-based model copyright protection device can be found in the limitations of the neuron-based model copyright protection method described above, and will not be repeated here. Each module in the above system can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module. It also achieves the advantages of reducing the interference and impact of watermark recognition on normal image classification and enhancing the concealment of watermark recognition.

[0076] In some embodiments of this application, an electronic device is also provided, comprising: at least one processor; and a memory connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which executes the aforementioned neuron-based model copyright protection method. Its internal structure diagram can be shown as follows. Figure 4 As shown. Figure 4 This diagram schematically illustrates the internal structure of an electronic device according to an embodiment of this application. The electronic device includes a processor A01, a network interface A02, a memory (not shown), and a database (not shown) connected via a system bus. The processor A01 provides computational and control capabilities. The memory includes internal memory A03 and a non-volatile storage medium A04. The non-volatile storage medium A04 stores an operating system B01, a computer program B02, and a database (not shown). The internal memory A03 provides an environment for the operation of the operating system B01 and the computer program B02 stored in the non-volatile storage medium A04. The network interface A02 is used for communication with external terminals via a network connection. When the computer program B02 is executed by the processor A01, it implements a model copyright protection method based on neuron screening.

[0077] Those skilled in the art will understand that Figure 4The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the electronic device to which the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0078] In one embodiment provided in this application, a machine-readable storage medium is provided, on which instructions are stored, which, when executed by a processor, cause the processor to be configured to perform the aforementioned model copyright protection method based on neuron screening.

[0079] In one embodiment provided in this application, a computer program product is provided, including a computer program that, when executed by a processor, implements the aforementioned model copyright protection method based on neuron screening.

[0080] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0081] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0082] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0083] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0084] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0085] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0086] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0087] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0088] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A model copyright protection method based on neuron screening, characterized by, The method includes: The CNN-based image classification model is divided into multiple neuron layers. The parameters to be solved in each neuron layer have weight coefficients. A first target loss function is constructed based on the classification loss, the magnitude of the parameters to be solved, and the weight coefficients. The image classification model using the first objective loss function is trained using a first training dataset containing image samples. During the training process, the weight coefficients are adjusted and a set of activated neurons is obtained. Obtaining the set of activated neurons during the training process includes: adding neurons whose output values ​​are not 0 and are not in fully connected layers from the image classification model during the training process to the set of activated neurons; dividing the CNN-based image classification model into non-fully connected layers and fully connected layers, wherein the fully connected layers are located in the last few layers of the image classification model. Obtain a second training dataset, which includes image sample pairs consisting of image samples and new images obtained by adding watermarked images to the image samples, wherein the category label of the new images is a preset fixed category, and construct a second objective loss function based on classification loss; The second training dataset is used to train an image classification model that uses the second objective loss function and has fixed neuron parameter values ​​in the set of activated neurons, resulting in a trained image classification model. The trained image classification model is verified by classifying an input image containing the watermark image into the preset fixed category. Constructing a first target loss function based on classification loss, the magnitude of the parameters to be solved, and the weight coefficients includes: using the magnitude of the parameters to be solved and the weight coefficients as the accumulated value of the parameters as a first part; using the cross-entropy loss function as a second part; and summing the first part and the second part to obtain the first target loss function. The first part, which uses the magnitude of the parameter to be solved and the weight coefficient as the cumulative value of the parameter, includes: using the second norm of the parameter to be solved as the magnitude of the parameter to be solved; obtaining the cumulative value of the current layer by weighted summation of the square of the magnitude of the parameter to be solved in the same neuron layer and the weight coefficient of the neuron layer; and obtaining the first part by summing the cumulative values ​​of the current layer of all neuron layers.

2. The method according to claim 1, characterized in that, The trained image classification model is validated by classifying an input image containing the watermark image into the preset fixed category, including: Multiple test images that do not belong to the preset fixed category and do not contain the watermark image are obtained to form a first verification set; A watermark image is added to each test image in the first verification set to obtain the second verification set; The second validation set is input into the image classification model to be validated to obtain the classification result corresponding to each input image in the second validation set. The percentage of classification results belonging to the preset fixed category is calculated among all classification results. When the proportion is higher than a preset ratio, it indicates that the image classification model to be verified is related to the watermark image used by the trained image classification model.

3. The method according to claim 1, characterized in that, The CNN-based image classification model is divided into multiple neuron layers, including: The CNN-based image classification model is divided into non-fully connected layers and fully connected layers, with the fully connected layers located in the last few layers of the image classification model. The non-fully connected layer is divided into multiple neuron layers.

4. The method according to claim 1, characterized in that, Adjusting the weight coefficients during training includes: In an image classification model, when the vector of all parameters to be solved is updated during training, the proportion of the number of parameters to be solved in each neuron layer whose values ​​are within a preset range is obtained to the total number of all parameters to be solved in that layer. When the ratio is less than the lower limit of the preset ratio threshold, the weight coefficient corresponding to the neuron layer is increased; When the ratio is greater than the upper limit of the preset ratio threshold, the weight coefficient corresponding to the neuron layer is reduced.

5. The method according to claim 1, characterized in that, The image samples in the image sample pairs in the second training dataset are selected from the first training dataset.

6. The method according to claim 1, characterized in that, Construct a second objective loss function based on the classification loss, including: The first part is obtained by taking the mean of the negative logarithms of the classification probabilities of new images in image sample pairs based on the image classification model. The second part is obtained based on the cross-entropy loss function of the classification results of image samples in image sample pairs by the image classification model; The second objective loss function is constructed based on the first and second parts.

7. The method according to claim 1, characterized in that, When training an image classification model using the first objective loss function with a first training dataset containing image samples, and when training an image classification model using the second objective loss function with a second training dataset and whose neuron parameter values ​​in the set of activated neurons are fixed, the Mini-batch SGD algorithm is used for iterative iteration.

8. An electronic device, characterized in that, include: At least one processor; A memory connected to the at least one processor; The memory stores instructions executable by the at least one processor, which executes the steps of the model copyright protection method based on neuron screening as described in any one of claims 1 to 7 by executing the instructions stored in the memory.

9. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When executed by a processor, the computer program / instruction implements the steps of the model copyright protection method based on neuron screening as described in any one of claims 1 to 7.

10. A computer program product, comprising a computer program / instructions, characterized in that, When executed by a processor, the computer program / instruction implements the steps of the model copyright protection method based on neuron screening as described in any one of claims 1 to 7.

Images