Model copyright protection method based on neuron screening and electronic equipment

Abstract

The embodiment of the invention provides a model copyright protection method based on neuron screening and electronic equipment, and relates to the technical field of digital copyright. The method comprises the following steps: dividing a CNN-based image classification model into a plurality of neuron layers, wherein a to-be-solved parameter in each neuron layer has a weight coefficient; training an image classification model adopting a first target loss function by adopting a first training data set containing image samples, and adjusting the weight coefficient in the training process to obtain an activated neuron set; obtaining a second training data set; training an image classification model which adopts the second target loss function and has fixed neuron parameter values in an activated neuron set by adopting the second training data set to obtain a trained image classification model; according to the implementation mode provided by the invention, the interference and influence of the watermark recognition function on the normal image classification function are reduced, and the concealment of the watermark recognition function is enhanced.

Description

Technical Field

[0001] The present application relates to the fields of digital copyright technology, deep learning technology, and image processing technology, and specifically to a model copyright protection method based on neuron screening, an electronic device, and a corresponding storage medium. Background Art

[0002] With the rapid development of artificial intelligence (AI), its applications are becoming increasingly widespread. It has been applied to a wide range of fields, including image analysis, speech recognition, text processing, intelligent recommendations, and security detection. Federated learning and large-scale model technologies have become cutting-edge AI hotspots. Copyright protection for AI systems is a key issue that needs to be addressed in the commercial application of AI technology. It is of great significance for protecting the intellectual property rights of AI technology, maintaining order in the AI ​​market, and promoting the promotion and application of AI technology. However, the current mainstream AI copyright protection technology directly trains the final model using watermarked samples along with the original samples as training samples. In most cases, watermarked samples interfere with the model's normal image classification function, and may even significantly reduce the model's normal image classification function and performance.

[0003] Existing technical solutions exist that use any global feature as the basis for watermarking. However, these solutions require manual design of a global feature quantization function, which is quite difficult in scenarios with a large number of image categories and model parameters. If the global feature quantization function is not designed properly, the watermarking effect will be significantly affected. Summary of the Invention

[0004] The purpose of the embodiments of the present application is to provide a model copyright protection method and electronic device based on neuron screening, which comprehensively adopts technologies such as hierarchical sparsification, neuron activation value screening, and model local parameter training to separate the training process of the model's normal image classification function from the training process of the watermark recognition function, so as to reduce the interference and influence of the watermark recognition function on the normal image classification function, enhance the concealment of the watermark recognition function, and at least solve some of the problems in the background technology.

[0005] In order to achieve the above-mentioned purpose, the present application provides a model copyright protection method based on neuron screening, which includes: dividing a CNN-based image classification model into multiple neuron layers, wherein the parameters to be solved in each neuron layer have weight coefficients, and constructing a first target loss function based on the classification loss, the size of the parameters to be solved and the weight coefficients; using a first training data set containing image samples to train the image classification model using the first target loss function, adjusting the weight coefficients during the training process and obtaining an activated neuron set; obtaining a second training data set, wherein the second training data set includes image sample pairs consisting of image samples and new images obtained by adding a watermark image to the image samples, the category label of the new image is a preset fixed category, and constructing a second target loss function based on the classification loss; using the second training data set to train the image classification model using the second target loss function and with fixed values ​​of neuron parameters in the activated neuron set to obtain a trained image classification model; the trained image classification model is verified by classifying the input image containing the watermark image into the preset fixed category when it is detected.

[0006] Preferably, the trained image classification model is verified by classifying the input image containing the watermark image into the preset fixed category when it is detected, including: obtaining multiple test images that do not belong to the preset fixed category and do not contain the watermark image to form a first verification set; adding a watermark image to each test image in the first verification set to obtain a second verification set; inputting the second verification set into the image classification model to be verified to obtain the classification result corresponding to each input image in the second verification set; counting the proportion of classification results classified into the preset fixed category in all classification results; when the proportion is higher than a preset ratio, it indicates that the image classification model to be verified has an association with the watermark image adopted by the trained image classification model.

[0007] Preferably, the CNN-based image classification model is divided into multiple neuron layers, including: dividing the CNN-based image classification model into non-fully connected layers and fully connected layers, and the fully connected layers are located in the last several layers of the image classification model; dividing the non-fully connected layers into multiple neuron layers.

[0008] Preferably, a first objective loss function is constructed based on the classification loss, the size of the parameter to be solved and the weight coefficient, including: taking the cumulative value of the parameter according to the size of the parameter to be solved and the weight coefficient as the first part; taking the cross entropy loss function as the second part; and summing the first part and the second part to obtain the first objective loss function.

[0009] Preferably, the first part is based on the size of the parameter to be solved and the weight coefficient as the cumulative value of the parameter, including: taking the second norm of the parameter to be solved as the size of the parameter to be solved; obtaining the cumulative value of this layer by weighted summation of the square of the size of the parameter to be solved in the same neuron layer and the weight coefficient of the neuron layer; and obtaining the first part by accumulating the cumulative values ​​of all neuron layers in this layer.

[0010] Preferably, the weight coefficient is adjusted during the training process, including: when the vector composed of all the parameters to be solved in the image classification model is updated during the training process, obtaining the ratio of the number of parameters to be solved in each neuron layer whose values ​​are within a preset range to the total number of all parameters to be solved in the layer; when the ratio is less than the lower limit of the preset ratio threshold, the weight coefficient corresponding to the neuron layer is adjusted upward; when the ratio is greater than the upper limit of the preset ratio threshold, the weight coefficient corresponding to the neuron layer is adjusted downward.

[0011] Preferably, obtaining the activated neuron set during the training process includes: adding neurons whose neuron output values ​​in the image classification model during the training process are not 0 and are not in the fully connected layer to the set to obtain the activated neuron set.

[0012] Preferably, the image samples in the image sample pairs in the second training data set are selected from the first training data set.

[0013] Preferably, a second target loss function is constructed based on the classification loss, including: obtaining a first part by calculating the average of the negative logarithm of the classification probability of the new image in the image sample pair according to the image classification model; obtaining a second part by calculating the cross entropy loss function of the classification results of the image samples in the image sample pair according to the image classification model; and constructing the second target loss function based on the first part and the second part.

[0014] Preferably, when using a first training data set containing image samples to train an image classification model that uses the first objective loss function, and when using the second training data set to train an image classification model that uses the second objective loss function and has fixed values ​​of neuron parameters in the activated neuron set, a Mini-batch SGD algorithm is used for cyclic iteration.

[0015] The present application also provides a model copyright protection device based on neuron screening, which includes: a training preparation module, which is used to divide the CNN-based image classification model into multiple neuron layers, and the parameters to be solved in each neuron layer have weight coefficients, and a first target loss function is constructed based on the classification loss, the size of the parameters to be solved and the weight coefficients; a first training module, which is used to use a first training data set containing image samples to train the image classification model using the first target loss function, adjust the weight coefficients during the training process, and obtain an activated neuron set; a watermark preparation module, which is used to obtain a second training data set, the second training data set including image sample pairs consisting of image samples and new images obtained by adding watermark images to the image samples, the category labels of the new images are preset fixed categories, and a second target loss function is constructed based on the classification loss; and a second training module, which is used to use the second training data set to train the image classification model using the second target loss function and with fixed values ​​of neuron parameters in the activated neuron set, to obtain a trained image classification model; the trained image classification model is verified by classifying the input image containing the watermark image into the preset fixed category when it is detected.

[0016] This application also provides an electronic device comprising: at least one processor; and a memory connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the at least one processor implements the aforementioned neuron screening-based model copyright protection method by executing the instructions stored in the memory.

[0017] The present application also provides a machine-readable storage medium having instructions stored thereon. When the instructions are executed by a processor, the processor is configured to execute the aforementioned model copyright protection method based on neuron screening.

[0018] The present application also provides a computer program product, comprising a computer program, which implements the aforementioned neuron screening-based model copyright protection method when executed by a processor.

[0019] The above technical solution has the following beneficial effects:

[0020] (1) This application separates the model training process for normal image classification tasks from the training process for watermark recognition functions. During the training process for normal image classification tasks, the parameter sparsity of each layer of the model is periodically quantitatively evaluated and controlled to ensure that each layer of the model can reserve enough redundant neurons to ensure that the model training for the watermark recognition function can be successfully completed.

[0021] (2) This application first trains the model for the normal image classification task, then fixes the neuron parameter values ​​related to the normal image classification task, and then trains the other unfixed parameters of the model for the watermark recognition task, which can reduce the negative impact of the added watermark recognition function on the normal image classification function of the model.

[0022] (3) The process of adding a watermark recognition function to the model provided in this application does not change the internal structure and number of parameters of the model. It is difficult to detect that the watermark recognition function has been added to the model from the external behavior and structure of the model, which greatly increases the concealment of the model watermark recognition function.

[0023] Other features and advantages of the embodiments of the present application will be described in detail in the subsequent detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the following detailed description, they are used to explain the embodiments of the present application but do not constitute a limitation on the embodiments of the present application. In the accompanying drawings:

[0025] Figure 1 Schematically showing the steps of the model copyright protection method based on neuron screening according to an embodiment of the present application;

[0026] Figure 2 Schematically shows a schematic diagram of the implementation process of the model copyright protection method based on neuron screening according to the embodiment of the present application;

[0027] Figure 3 Schematically shows a structural diagram of a model copyright protection device based on neuron screening according to an embodiment of the present application;

[0028] Figure 4 The internal structure of an electronic device according to an embodiment of the present application is schematically shown. DETAILED DESCRIPTION

[0029] The following describes the specific implementation of the embodiment of the present application in detail with reference to the accompanying drawings. It should be understood that the specific implementation described herein is only used to illustrate and explain the embodiment of the present application and is not intended to limit the embodiment of the present application.

[0030] Figure 1 The following schematically shows the steps of the copyright protection method of the model based on neuron screening according to the embodiment of the present application. Figure 1 As shown, a model copyright protection method based on neuron screening includes:

[0031] S01, dividing the CNN-based image classification model into multiple neuron layers, wherein the parameters to be solved in each neuron layer have weight coefficients, and constructing a first objective loss function based on the classification loss, the size of the parameters to be solved, and the weight coefficients;

[0032] S02, using a first training data set containing image samples to train an image classification model using the first target loss function, adjusting the weight coefficient and obtaining an activated neuron set during the training process;

[0033] S03. Obtain a second training data set, where the second training data set includes image sample pairs consisting of image samples and new images obtained by adding a watermark image to the image samples, where the category labels of the new images are preset fixed categories, and construct a second target loss function based on the classification loss;

[0034] S04. Use the second training data set to train an image classification model that uses the second objective loss function and has fixed values ​​of neuron parameters in the activated neuron set to obtain a trained image classification model; the trained image classification model is verified by classifying the input image containing the watermark image into the preset fixed category when it is detected.

[0035] In step S01, the weight coefficient can be set based on the neuron layer, that is, the weight coefficient of the parameters to be solved in the same neuron layer is the same. The weight coefficient is periodically monitored and adjusted during the training process of step S02, thereby reducing the risk of watermark recognition function training failure caused by excessive differences in parameter sparsity distribution between model layers.

[0036] The watermark image in step S03 is used to provide a specific image identification, which can be one or more of a company logo, a company name in Chinese, a company name in English, an image of a specific shape, a specific character string, and the like.

[0037] In step S04, all the parameters to be solved in the image classification model are divided into: fixed-value parameters and unfixed-value parameters to be solved according to the activated neuron set. The training process in step S04 can only change the unfixed-value parameters to be solved, thereby avoiding negative impact on the parameters of the neurons in the activated neuron set.

[0038] Those skilled in the art will also appreciate that the preparation and acquisition of the second training dataset in step S03 does not use the parameters in the previous steps and therefore does not have a strict execution location. It can be performed before step S04. This is only for the purpose of text description. For example, steps S02 and S03 can be performed simultaneously, or step S03 can be performed before step S02, that is, the second training dataset is prepared and the second objective loss function is constructed first. Step S03 can also be prepared and acquired simultaneously with the first training dataset, or before or after acquiring the first training dataset.

[0039] The CNN-based image classification models in this embodiment and subsequent embodiments include but are not limited to AlexNet, VGG, GoogLeNet, ResNet, FPN, DenseNet, etc.

[0040] Through the above implementation method, the internal structure and number of parameters of the model can be kept unchanged. It is difficult to find out from the external behavior and structure of the model that the watermark recognition function has been added to the model, which greatly increases the concealment of the model watermark recognition function.

[0041] In some optional embodiments of the present application, the trained image classification model is verified by classifying the input image containing the watermark image into the preset fixed category when it is detected, including: obtaining multiple test images that do not belong to the preset fixed category and do not contain the watermark image to form a first verification set; adding a watermark image to each test image in the first verification set to obtain a second verification set; inputting the second verification set into the image classification model to be verified to obtain the classification result corresponding to each input image in the second verification set; counting the proportion of classification results classified as the preset fixed category in all classification results; when the proportion is higher than a preset ratio, it indicates that the image classification model to be verified has an association relationship with the watermark image used by the trained image classification model. The aforementioned association relationship includes but is not limited to: the image classification model to be verified is marked by the watermark image, the copyright of the image classification model to be verified belongs to the organization or individual that provides the watermark image, and the owner of the image classification model to be verified is related to the information in the watermark image. Specifically, for any model M′ to be verified for copyright, prepare multiple images to be classified that do not belong to the Kth category and are not watermarked. The above images constitute the image set V, which is the first verification set. Add the watermark image X to the image samples in the first verification set. w After synthesizing new images, the first validation set becomes the second validation set V'. Count the proportion of all images in the second validation set that are predicted by the model M' as the Kth category. At this time, P is the lower limit of the proportion of all images in V' that are identified as the Kth category. If it is lower than P, it means that no watermark is embedded. If it exceeds P, it can be judged that the model M' has added a watermark X.w , the copyright of the model M′ belongs to the organization or individual that provides the watermark. Where P∈(0,1], P represents the confidence of the model being embedded in the watermark. The Kth category is the preset fixed category.

[0042] In some optional embodiments of the present application, the CNN-based image classification model is divided into multiple neuron layers, including: dividing the CNN-based image classification model into non-fully connected layers and fully connected layers, wherein the fully connected layers are located in the last several layers of the image classification model; and dividing the non-fully connected layers into multiple neuron layers. Assume that M is a CNN-based image classification model, all hidden layer neurons of model M use ReLU activation function, the last several layers of model M are fully connected layers, model M is divided into L neuron layers, and all parameters to be solved in the lth layer of model M constitute a set Θ l ,Θ l The mth parameter to be solved is denoted as Set Θ l The cardinality of |Θ l This implementation divides the model into two parts: a non-fully connected layer and a fully connected layer. Different training strategies are used for the parameters of the two parts. This maximizes the classification accuracy of the fully connected layer while reducing the negative interference of watermark image features on normal image features.

[0043] In some optional embodiments of the present application, a first target loss function is constructed based on the classification loss, the size of the parameter to be solved and the weight coefficient, including: taking the size of the parameter to be solved and the weight coefficient as the cumulative value of the parameter as the first part; taking the cross entropy loss function as the second part; summing the first part and the second part to obtain the first target loss function. And further, taking the second norm of the parameter to be solved as the size of the parameter to be solved; taking the square of the size of the parameter to be solved in the same neuron layer and the weight coefficient of the neuron layer by weighted summation to obtain the cumulative value of this layer; and summing the cumulative values ​​of this layer of all neuron layers to obtain the first part. D is the first training sample image set of model M, and the first target loss function L adopted in the training process of model M is D (θ) is shown in the following formula. θ is the vector of all parameters to be solved for the model M, 1≤l≤L, 1≤m≤|Θ l |, l, L, m are positive integers.

[0044]

[0045] Among them, 1(x) is the indicator function, specifically:

[0046]

[0047] Among them, B is the number of sample images in each Batch, C is the total number of sample image categories contained in the training set D and the test set T, and y i is the category label number of the i-th sample image in a Batch of the training set D, The probability that the model M predicts the i-th sample image as the k-th category. l is the weight coefficient of the parameters to be solved in the lth layer of model M. Is the square of the second norm of vector x. B, C, y i , i, k are all positive integers, 1≤i≤B, 1≤k≤C, 1≤y i ≤C,α l >0. The first objective loss function mainly introduces the size of the parameter to be solved and the weight coefficient as parameters, through The total size of the loss is calculated for The value of is generally suppressed.

[0048] In some embodiments of the present application, the weight coefficient is adjusted during the training process, including: when the vector composed of all the parameters to be solved in the image classification model is updated during the training process, the ratio of the number of parameters to be solved in each neuron layer whose values ​​are within a preset range to the total number of all parameters to be solved in the layer is obtained; when the ratio is less than the lower limit of the preset ratio threshold, the weight coefficient corresponding to the neuron layer is adjusted upward; when the ratio is greater than the upper limit of the preset ratio threshold, the weight coefficient corresponding to the neuron layer is adjusted downward. Specifically, after the rth iterative training cycle of model M (each time the parameter θ of model M is updated, it indicates the end of an iterative training cycle), the ratio of the number of parameters to be solved in each neuron layer of model M whose values ​​are not 0 to the total number of all parameters to be solved in the layer is calculated, and S l It represents the ratio of the number of parameters to be solved whose values ​​are not 0 in the first layer of model M to the total number of all parameters to be solved in this layer. l MIN , then increase α l If S l >S MAX , then reduce α l If the model training process does not meet the end condition (the model parameter values ​​converge or r exceeds the preset threshold), then set r = r + 1 and start the next iterative training cycle. MIN ∈[0,1], S MAX ∈[0,1], S MAX ≥S MIN ​This embodiment periodically quantitatively evaluates and controls the parameter sparsity of each layer of the model during the training process for normal image classification tasks, ensuring that each layer of the model can reserve enough redundant neurons to ensure that the model training for the watermark recognition function can be successfully completed.

[0049] In some embodiments of the present application, obtaining the set of activated neurons during training includes: adding neurons in the image classification model whose neuron output values ​​are not 0 and are not in a fully connected layer during training to a set to obtain the set of activated neurons. Specifically, for each image sample X in a dataset D, X is input into a model M, and neurons in the model M whose neuron output values ​​are not 0 and are not in a fully connected layer are recorded, and these neurons are added to a set A, which is the set of activated neurons.

[0050] In some embodiments of the present application, the image samples in the image sample pairs in the second training data set are selected from the first training data set. This embodiment defines the relationship between the second training data set and the first training data set. Assume that X w is a watermark image whose size is smaller than that of the training sample images in the training dataset D. The watermark image can contain one or more of the following: company logo, company name in Chinese or English, image of specific shape, specific character string, etc. For each image sample X in D, w Add it to the image sample X to synthesize a new image X′. All image sample pairs (X, X′) constitute the image set D′, which is the second training dataset. Set the category labels of all X′ in D′ to K∈[1,C], where K is a positive integer constant.

[0051] In some embodiments of the present application, a second objective loss function is constructed based on the classification loss, including: obtaining a first part by calculating the average of the logarithmic probability of the classification of the new image in the image sample pair according to the image classification model; obtaining a second part by performing a cross entropy loss function on the classification results of the image samples in the image sample pair according to the image classification model; and constructing the second objective loss function based on the first part and the second part. The second objective loss function L used by the training model M is D′ (θ) is shown in the following formula.

[0052]

[0053] Among them, y j is the category label number of X in the j-th sample image pair of the image set D′, It is the probability that the model M predicts X′ in the j-th image sample pair in a Batch of the image set D′ as the K-th category, It is the probability that the model M predicts X in the j-th image sample pair in a batch of the image set D′ as the k-th category. 1≤j≤B, j is a positive integer.

[0054] The image set D′ is used as the training sample set, the parameter values ​​of the neurons in the set A are fixed, each batch contains B image sample pairs, a total of 2B image samples, and the Mini-Batch SGD algorithm is used to iteratively train all the parameters of the model M whose values ​​are not fixed.

[0055] In some embodiments of the present application, when a first training dataset containing image samples is used to train an image classification model using the first objective loss function, and when a second training dataset is used to train an image classification model using the second objective loss function and with fixed values ​​of neuron parameters in the activated neuron set, a Mini-batch SGD algorithm is used for cyclic iteration. Compared with other learning algorithms, mini-batch SGD has the following advantages: (1) lower computational cost because only a small batch of gradients is calculated each time; (2) faster convergence to the global optimal solution because each batch may contain some samples far from the local optimal solution; (3) less memory consumption for large-scale datasets because only a small batch of samples is loaded each time. This embodiment limits the update method of model parameters and improves the efficiency of model training.

[0056] The model copyright protection method based on neuron screening in the above implementation mode can not change the internal structure and number of parameters of the model, and it is difficult to find out from the external behavior and structure of the model that the watermark recognition function has been added to the model, thereby greatly increasing the concealment of the model watermark recognition function.

[0057] Figure 2 The following schematically shows the implementation process of the model copyright protection method based on neuron screening according to the embodiment of the present application. Figure 2 As shown, the specific implementation process of the model copyright protection method based on neuron screening is as follows:

[0058] (1) Assume that M is a CNN-based image classification model (such as AlexNet, VGG, GoogLeNet, ResNet, FPN, DenseNet, etc.). All hidden layer neurons of model M use the ReLU activation function. The last several layers of model M are fully connected layers, and model M is divided into L neuron layers. D is the training sample image set of model M (for example, containing 128,000 images, 10 categories, and image size is 224×224×3). Based on the training sample set D, the parameters of model M are trained using the Mini-batch SGD algorithm in a cyclic iterative manner.

[0059] (2) After the rth iteration training cycle of model M (each update of the parameter θ of model M indicates the end of an iteration training cycle), the proportion of the number of unsolved parameters with non-zero values ​​in each neuron layer of model M to the total number of all unsolved parameters in this layer is calculated, and S is used to calculate the ratio. l It represents the ratio of the number of parameters to be solved whose values ​​are not 0 in the first layer of model M to the total number of all parameters to be solved in this layer. l MIN , then increase α l If S l >S MAX , then reduce α l If the model training process does not meet the end condition (model parameter values ​​converge or r exceeds 200), r=r+1 is set to start the next iterative training cycle.

[0060] (3) For each image sample X in the dataset D, input X into the model M, record the neurons in the model M whose neuron output value is not 0 and is not in the fully connected layer, and add the above neurons to the set A

[0061] (4) Assume X w The watermark image is a Topsec string. For each image sample X in D, w Add to the image sample X to synthesize a new image X′, and all image sample pairs (X, X′) constitute the image set D′. Set the category labels of all X′ in D′ to 3.

[0062] (5) The image set D′ is used as the training sample set, and the parameter values ​​of the neurons in the set A are fixed. Each batch contains B = 128 image sample pairs, for a total of 256 image samples. The Mini-Batch SGD algorithm is used to iteratively train all the parameters of the model M whose values ​​are not fixed.

[0063] (6) For any model M′ to be verified for copyright, prepare 100 images to be classified that do not belong to the third category and are not watermarked. The above images constitute the image set V. Add the watermark image X to the image samples in the set V. w After synthesizing the new image, the image set is recorded as V'. If the proportion of new images in V' predicted by model M' as the third category is higher than the preset ratio P, where P = 90%, it can be determined that the model M' has added the watermark X w The copyright of the model M′ belongs to the organization or individual that provides the watermark.

[0064] ​It can be seen from the above implementation methods that the present application has the advantage of not changing the internal structure and number of parameters of the model, and it is difficult to find out from the external behavior and structure of the model that the watermark recognition function has been added to the model, which greatly increases the concealment of the model watermark recognition function.

[0065] Based on the same inventive concept, this application also provides a model copyright protection device based on neuron screening. Figure 3 The schematic diagram of the structure of the model copyright protection device based on neuron screening according to the embodiment of the present application is shown. Figure 3 As shown, the device includes: a training preparation module, which is used to divide the CNN-based image classification model into multiple neuron layers, and the parameters to be solved in each neuron layer have weight coefficients, and a first target loss function is constructed based on the classification loss, the size of the parameters to be solved and the weight coefficients; a first training module, which is used to use a first training data set containing image samples to train the image classification model using the first target loss function, adjust the weight coefficients during the training process, and obtain an activated neuron set; a watermark preparation module, which is used to obtain a second training data set, the second training data set including image sample pairs consisting of image samples and new images obtained by adding watermark images to the image samples, the category labels of the new images are preset fixed categories, and a second target loss function is constructed based on the classification loss; and a second training module, which is used to use the second training data set to train the image classification model using the second target loss function and with fixed values ​​of neuron parameters in the activated neuron set, to obtain a trained image classification model; the trained image classification model is verified by classifying the input image containing the watermark image into the preset fixed category when it is detected.

[0066] In some optional embodiments of the present application, the trained image classification model is verified by classifying the input image containing the watermark image into the preset fixed category when it is detected, including: obtaining multiple test images that do not belong to the preset fixed category and do not contain the watermark image to form a first verification set; adding a watermark image to each test image in the first verification set to obtain a second verification set; inputting the second verification set into the image classification model to be verified to obtain the classification result corresponding to each input image in the second verification set; counting the proportion of classification results classified as the preset fixed category in all classification results; when the proportion is higher than a preset ratio, it indicates that the image classification model to be verified has an association with the watermark image adopted by the trained image classification model.

[0067] In some optional embodiments of the present application, the CNN-based image classification model is divided into multiple neuron layers, including: dividing the CNN-based image classification model into non-fully connected layers and fully connected layers, and the fully connected layers are located in the last several layers of the image classification model; dividing the non-fully connected layers into multiple neuron layers.

[0068] In some optional embodiments of the present application, a first objective loss function is constructed based on the classification loss, the size of the parameter to be solved and the weight coefficient, including: taking the cumulative value of the parameter according to the size of the parameter to be solved and the weight coefficient as the first part; taking the cross entropy loss function as the second part; and summing the first part and the second part to obtain the first objective loss function.

[0069] In some optional embodiments of the present application, the first part is based on the size of the parameter to be solved and the weight coefficient as the cumulative value of the parameter, including: taking the second norm of the parameter to be solved as the size of the parameter to be solved; obtaining the cumulative value of this layer by weighted summation of the square of the size of the parameter to be solved in the same neuron layer and the weight coefficient of the neuron layer; and obtaining the first part by accumulating the cumulative values ​​of this layer of all neuron layers.

[0070] In some optional embodiments of the present application, the weight coefficient is adjusted during the training process, including: when the vector consisting of all the parameters to be solved in the image classification model is updated during the training process, obtaining the ratio of the number of parameters to be solved in each neuron layer whose values ​​are within a preset range to the total number of all parameters to be solved in the layer; when the ratio is less than the lower limit of the preset ratio threshold, the weight coefficient corresponding to the neuron layer is adjusted upward; when the ratio is greater than the upper limit of the preset ratio threshold, the weight coefficient corresponding to the neuron layer is adjusted downward.

[0071] In some optional embodiments of the present application, obtaining an activated neuron set during training includes: adding neurons whose neuron output values ​​in the image classification model during training are not 0 and are not in the fully connected layer to the set to obtain the activated neuron set.

[0072] In some optional implementations of the present application, the image samples in the image sample pairs in the second training data set are selected from the first training data set.

[0073] In some optional embodiments of the present application, a second target loss function is constructed based on the classification loss, including: obtaining a first part by performing logarithmic calculation on the average of the classification probability of the new image in the image sample pair according to the image classification model; obtaining a second part by performing a cross-entropy loss function on the classification results of the image samples in the image sample pair according to the image classification model; and constructing the second target loss function based on the first part and the second part.

[0074] In some optional embodiments of the present application, when using a first training data set containing image samples to train an image classification model that uses the first objective loss function, and when using the second training data set to train an image classification model that uses the second objective loss function and has fixed values ​​of neuron parameters in the activated neuron set, a Mini-batch SGD algorithm is used for cyclic iteration.

[0075] The specific definitions of the various functional modules in the aforementioned neuron-based model copyright protection device can be found in the definitions of the neuron-based model copyright protection method described above and will not be repeated here. Each module in the aforementioned system can be implemented in whole or in part through software, hardware, or a combination thereof. Each of these modules can be embedded in or independent of a processor in a computer device in hardware form, or stored in a computer device memory in software form, allowing the processor to call and execute the corresponding operations of each module. This also achieves the advantage of reducing the interference and impact of the watermark recognition function on the normal image classification function, thereby enhancing the concealment of the watermark recognition function.

[0076] In some embodiments of the present application, an electronic device is further provided, comprising: at least one processor; a memory connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the at least one processor executes the aforementioned model copyright protection method based on neuron screening. Its internal structure diagram can be as follows: Figure 4 shown. Figure 4 The internal structure diagram of an electronic device according to an embodiment of the present application is schematically shown. The electronic device includes a processor A01, a network interface A02, a memory (not shown in the figure) and a database (not shown in the figure) connected by a system bus. Among them, the processor A01 of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes an internal memory A03 and a non-volatile storage medium A04. The non-volatile storage medium A04 stores an operating system B01, a computer program B02 and a database (not shown in the figure). The internal memory A03 provides an environment for the operation of the operating system B01 and the computer program B02 in the non-volatile storage medium A04. The network interface A02 of the electronic device is used to communicate with an external terminal through a network connection. When the computer program B02 is executed by the processor A01, a model copyright protection method based on neuron screening is implemented.

[0077] Those skilled in the art will understand that Figure 4The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the electronic device to which the solution of the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0078] In one embodiment provided in the present application, a machine-readable storage medium is provided, on which instructions are stored. When the instructions are executed by a processor, the processor is configured to execute the aforementioned model copyright protection method based on neuron screening.

[0079] In one embodiment provided in the present application, a computer program product is provided, including a computer program, which implements the aforementioned model copyright protection method based on neuron screening when executed by a processor.

[0080] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0081] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0082] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0083] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0084] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0085] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0086] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0087] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0088] The above are merely embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. A model copyright protection method based on neuron screening, characterized in that: The method includes: Dividing the CNN-based image classification model into multiple neuron layers, wherein the parameters to be solved in each neuron layer have weight coefficients, and constructing a first objective loss function based on the classification loss, the size of the parameters to be solved, and the weight coefficients; Using a first training data set containing image samples to train an image classification model using the first target loss function, adjusting the weight coefficient and obtaining an activated neuron set during the training process; Obtaining a second training data set, the second training data set including image sample pairs consisting of image samples and new images obtained by adding a watermark image to the image samples, the category labels of the new images being preset fixed categories, and constructing a second target loss function based on the classification loss; The second training data set is used to train an image classification model that adopts the second objective loss function and has fixed values ​​of neuron parameters in the activated neuron set to obtain a trained image classification model; the trained image classification model is verified by classifying the input image containing the watermark image into the preset fixed category when it is detected.

2. The method according to claim 1, characterized in that The trained image classification model is verified by classifying an input image containing the watermark image into the preset fixed category when detecting the input image, including: Acquire multiple test images that do not belong to the preset fixed category and do not contain the watermark image to form a first verification set; Adding a watermark image to each test image in the first verification set to obtain a second verification set; Inputting the second verification set into the image classification model to be verified, and obtaining a classification result corresponding to each input image in the second verification set; Counting the proportion of classification results classified into the preset fixed category in all classification results; When the proportion is higher than a preset ratio, it indicates that the image classification model to be verified has an association relationship with the watermark image used by the trained image classification model.

3. The method according to claim 1, characterized in that The CNN-based image classification model is divided into multiple neuron layers, including: Dividing the CNN-based image classification model into non-fully connected layers and fully connected layers, wherein the fully connected layers are located in the last several layers of the image classification model; The non-fully connected layer is divided into multiple neuron layers.

4. The method according to claim 1, wherein Constructing a first objective loss function based on the classification loss, the size of the parameter to be solved, and the weight coefficient includes: The first part is the cumulative value of the parameter according to the size of the parameter to be solved and the weight coefficient; Use cross entropy loss function as the second part; The first part and the second part are summed to obtain the first objective loss function.

5. The method according to claim 4, characterized in that The first part is based on the size of the parameter to be solved and the weight coefficient as the cumulative value of the parameter, including: Taking the second norm of the parameter to be solved as the size of the parameter to be solved; The cumulative value of this layer is obtained by weighted summing the square of the parameter to be solved in the same neuron layer and the weight coefficient of the neuron layer; The first part is obtained by adding up the current layer cumulative values ​​of all neuron layers.

6. The method according to claim 1, characterized in that Adjusting the weight coefficients during training includes: When a vector of all parameters to be solved in the image classification model is updated during training, the ratio of the number of parameters to be solved whose values ​​are within a preset range to the total number of all parameters to be solved in each neuron layer is obtained; When the ratio is less than the lower limit of the preset ratio threshold, the weight coefficient corresponding to the neuron layer is increased; When the ratio is greater than a preset ratio threshold upper limit, the weight coefficient corresponding to the neuron layer is lowered.

7. The method according to claim 1, characterized in that During the training process, the activated neuron set is obtained, including: The neurons whose neuron output values ​​in the image classification model during the training process are not 0 and are not in the fully connected layer are added to the set to obtain the activated neuron set.

8. The method according to claim 1, characterized in that The image samples in the image sample pairs in the second training dataset are selected from the first training dataset.

9. The method according to claim 1, characterized in that The second objective loss function is constructed based on the classification loss, including: The first part is obtained by calculating the average of the negative logarithm of the classification probability of the new image in the image sample pair according to the image classification model; The second part is obtained according to the cross entropy loss function of the classification results of the image samples in the image sample pairs by the image classification model; The second objective loss function is constructed according to the first part and the second part.

10. The method according to claim 1, characterized in that When using a first training data set containing image samples to train an image classification model that uses the first objective loss function, and when using the second training data set to train an image classification model that uses the second objective loss function and has fixed values ​​of neuron parameters in the activated neuron set, a Mini-batch SGD algorithm is used for cyclic iteration.

11. An electronic device, characterized in that: include: at least one processor; a memory connected to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the at least one processor implements the steps of the model copyright protection method based on neuron screening as described in any one of claims 1 to 10 by executing the instructions stored in the memory.

12. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the model copyright protection method based on neuron screening as described in any one of claims 1 to 10 are implemented.

13. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the model copyright protection method based on neuron screening as described in any one of claims 1 to 10 are implemented.

Images