Machine learning image classification model ownership verification method and system

Abstract

The application provides a machine learning image classification model ownership verification method and system, comprising the following steps: S1, generating a trigger sample set; S2, fine-tuning the picture classification model by using a model backdoor training technology, so that the fine-tuned picture classification model produces an incorrect prediction result that meets a preset requirement for the trigger sample set while maintaining normal prediction function; S3, generating a new trigger sample based on a trigger pattern in the trigger sample set, and inputting the new trigger sample into the fine-tuned picture classification model to complete forward reasoning; and S4, verifying the correctness of the new trigger sample generation process and the reasoning process of the fine-tuned picture classification model for the new trigger sample.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer artificial intelligence security, in particular to a machine learning image classification model ownership verification method and system, and more particularly to a machine learning image classification model ownership verification method and system based on zero-knowledge proof technology. BACKGROUND

[0002] The prior art needs to submit a subset of trigger samples as model ownership evidence to a trusted third-party institution for model ownership verification, and relies on the third party to check the evidence and make a model ownership arbitration. However, in actual application, there is no completely trusted third-party institution, so the submission of model ownership evidence may expose the trigger samples, which may further trigger attacks on the model watermark, including removing the watermark or re-watermarking the model watermark based on the common characteristics of the exposed trigger samples. These attacks will cause the model watermark to fail, including being unable to produce specific and incorrect prediction results for samples in the trigger set, or producing specific and incorrect prediction results for non-trigger samples, ultimately failing to effectively verify the ownership of the model.

[0003] On the other hand, some existing technical solutions suggest embedding multiple backdoors as watermark information in the target model using multiple disjoint trigger sample sets, and submitting only one sample set as model ownership arbitration evidence at each model ownership verification, i.e., exposing one watermark information, to support a limited number of ownership verification processes. However, in practice, embedding too many backdoors in the target model will affect the normal prediction ability of the model, so this solution cannot support an arbitrary number of model ownership verification processes, and is not very practical. SUMMARY

[0004] In view of the defects in the prior art, the purpose of the present application is to provide a machine learning image classification model ownership verification method and system.

[0005] According to the machine learning image classification model ownership verification method provided by the present application, the following steps are included:

[0006] Step S1: generating a trigger sample set;

[0007] Step S2: fine-tuning the picture classification model using model backdoor training technology, so that the fine-tuned picture classification model produces incorrect prediction results that meet the preset requirements for the trigger sample set while maintaining normal prediction functions;

[0008] Step S3: generating a new trigger sample based on the trigger pattern in the trigger sample set, and inputting the new trigger sample into the fine-tuned picture classification model to complete forward reasoning;

[0009] Step S4: verifying the correctness proof of the new trigger sample generation process and the inference process of the fine-tuned picture classification model on the new trigger sample.

[0010] Preferably, the step S1 comprises: superimposing the trigger pattern on a subset of the training set of the picture classification model to obtain a set of trigger samples.

[0011] The subset of the training set of the picture classification model comprises: setting a subset size ratio, and randomly selecting samples from each category of the training set as the training set subset according to the set subset size ratio;

[0012] The superimposition of the trigger pattern on the subset of the training set of the picture classification model comprises: representing each pixel point in the two pictures in the form of RGB three channels, then adding the values of the three channels at the corresponding positions as the superimposition result, and setting the superimposition result to 255 if the result is greater than the threshold value 255.

[0013] Preferably, the step S3 comprises:

[0014] Step S3.1: superimposing the trigger pattern in the set of trigger samples on a clean training set picture meeting the preset requirements to generate a new trigger sample;

[0015] Step S3.2: inputting the new trigger sample into the fine-tuned picture classification model, and completing the forward inference process by using the fine-tuned picture classification model to obtain the prediction result of the model.

[0016] Preferably, the step S4 comprises: verifying the correctness proof of the new trigger sample generation process and the inference process of the fine-tuned picture classification model on the new trigger sample.

[0017] Step S4.1: modeling the new trigger sample generation process and the inference process of the fine-tuned picture classification model on the new trigger sample into a zero-knowledge proof circuit;

[0018] Step S4.2: using a proof algorithm in a zero-knowledge proof protocol to prove the correctness of the zero-knowledge proof circuit and generate a zero-knowledge proof.

[0019] Preferably, the step S4.2 comprises: calling the Prove(·) algorithm in the zero-knowledge proof protocol PlonK to prove the correctness of the zero-knowledge proof circuit C constructed by using the PLONKish language and generate a zero-knowledge proof π = Prove(C, w, t); wherein w represents the pixel value of the trigger pattern; t represents the pixel value of the sample in the training set subset and the parameter information of the fine-tuned picture classification model.

[0020] The machine learning image classification model ownership verification system provided by the application comprises:

[0021] Module M1: generating a trigger sample set;

[0022] Module M2: fine-tuning the picture classification model by using a model backdoor training technology, so that the fine-tuned picture classification model produces an incorrect prediction result that meets preset requirements for the trigger sample set while maintaining normal prediction functions;

[0023] Module M3: generating a new trigger sample based on the trigger pattern in the trigger sample set and inputting the new trigger sample into the fine-tuned picture classification model to complete forward reasoning;

[0024] Module M4: verifying the correctness of the generation process of the new trigger sample and the reasoning process of the fine-tuned picture classification model for the new trigger sample.

[0025] Preferably, the module M1 comprises: superimposing the trigger pattern on a training set subset of the picture classification model to obtain the trigger sample set;

[0026] The training set subset of the picture classification model comprises: setting a subset size ratio and randomly selecting samples from each category of the training set as the training set subset according to the set subset size ratio;

[0027] The superimposition of the trigger pattern on the training set subset of the picture classification model comprises: representing each pixel point in the two pictures in the form of RGB three channels, then adding the values of the three channels at the corresponding positions as the superimposition result, and setting the superimposition result to 255 if the result is greater than the threshold value 255.

[0028] Preferably, the module M3 comprises:

[0029] Module M3.1: superimposing the trigger pattern in the trigger sample set on a clean training set picture that meets preset requirements to generate a new trigger sample;

[0030] Module M3.2: inputting the new trigger sample into the fine-tuned picture classification model and completing the forward reasoning process by using the fine-tuned picture classification model to obtain the prediction result of the model.

[0031] Preferably, the module M4 comprises: verifying the correctness of the generation process of the new trigger sample and the reasoning process of the fine-tuned picture classification model for the new trigger sample.

[0032] Module M4.1: modeling the generation process of the new trigger sample and the reasoning process of the fine-tuned picture classification model for the new trigger sample into a zero-knowledge proof circuit;

[0033] Module M4.2: using a proof algorithm in a zero-knowledge proof protocol, proving the correctness of the zero-knowledge proof circuit, and generating a zero-knowledge proof.

[0034] Preferably, the module M4.2 includes: calling the Prove(·) algorithm in the zero-knowledge proof protocol PlonK, proving the correctness of the zero-knowledge proof circuit C constructed by the PLONKish language, and generating a zero-knowledge proof π = Prove(C, w, t); wherein w represents the pixel value of the trigger pattern; t represents the pixel value of the sample in the training set subset and the parameter information of the fine-tuned picture classification model.

[0035] Compared with the prior art, the present application has the following beneficial effects:

[0036] 1. The zero-knowledge proof technology can generate a correctness proof of the operation relationship without revealing the key input information, and the model ownership verification can be completed in the case of exposing the trigger sample and watermark information, thereby resisting the watermark removal attack and watermarking attack against the model watermark and supporting the model ownership verification process for any number of times.

[0037] 2. Compared with the multi-party secure computing scheme, the zero-knowledge proof technology does not require the prover and the verifier to be online at the same time, and is more flexible in application.

[0038] 3. The present application proposes a robust watermarking scheme that does not expose the trigger sample and watermark information when used for model ownership verification, thereby resisting the watermark removal attack and watermarking attack against the model watermark and supporting the model ownership verification process for any number of times.

[0039] 4. The present application can expose the watermark information in the verification process, and resist the potential watermark removal attack and watermarking attack. BRIEF DESCRIPTION OF DRAWINGS

[0040] Other features, objects and advantages of the present application will become more apparent from the following detailed description of non-limiting embodiments with reference to the following drawings:

[0041] Figure 1 Machine learning image classification model ownership verification method flowchart. DETAILED DESCRIPTION

[0042] The present application will be described in detail below with reference to specific embodiments. The following embodiments will help those skilled in the art to further understand the present application, but do not limit the present application in any form. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present application. These all belong to the protection scope of the present application.

[0043] Example 1

[0044] According to the machine learning image classification model ownership verification method provided by the application, a trigger set is generated, a watermark is embedded in a target model, a model watermark is detected and a proof is generated, and the ownership of the model is verified and arbitrated. Figure 1 As shown in the figure, it includes: a trigger set generating step, a watermark embedding step for a target model, a model watermark detecting and proof generating step, and a model ownership verifying and arbitrating step.

[0045] The step of generating a trigger set: a previously selected trigger pattern needs to be superimposed on a subset of the machine learning model training set to make a trigger sample set.

[0046] More specifically, the superimposition process of the two pictures: each pixel point in the two pictures is represented in RGB three-channel form, and then the values of the three channels at the corresponding positions are added as the superimposition result. If the result is greater than the threshold value 255, the result is set to 255.

[0047] A fixed subset size ratio of 10% is set, and 10% of the samples from each category are randomly selected as a subset for making a trigger sample set.

[0048] The trigger pattern is a picture with the same size as the training set, except that the RGB pixel value of the 3*3 area in the lower left corner is (255, 0, 0), and the RGB pixel value of the other areas is (255, 255, 255).

[0049] The step of embedding a watermark in the target model: it is necessary to use model backdoor training technology to fine-tune the target model, so that it produces a specific, incorrect prediction result for the trigger sample set while maintaining normal prediction function; in this embodiment, the target model is an image classification model.

[0050] The step of detecting the model watermark and generating the proof: first, a previously selected trigger pattern needs to be superimposed on a clean training set picture to make a new trigger sample, then the trigger sample is used as the model input, and the forward inference process is completed using the target model to obtain the prediction result of the model, finally the trigger sample generation process and the inference process of the model on the trigger sample are modeled as a zero-knowledge proof circuit, using the proof algorithm in the zero-knowledge proof protocol, the correctness proof of the above two processes is generated without exposing the trigger pattern, and the correctness proof is submitted to a third-party institution.

[0051] Wherein, the modeling zero-knowledge proof circuit: using PLONKish circuit arithmetic language, the operation relationship to be verified defined on the finite field F is converted into the basic constraint form s of PLONKish L ·x L +s R ·x R +sO ·

[0052] x O +s M ·x L ·x R +s C =0, where s L ,s R ,s O ,s M ,s C ∈F represents a selector, which can be used to specify the type of operation gate to be used, x L ,x R ,x O ∈F represent the left input, right input, and output values ​​of the constraint, respectively. For constraints of the form x L +x R =x O The addition gate can be set by s L =s R =1,s O =-1,s M =s C =0 is achieved. For forms of x L ·x R =x O The multiplication gate can be set by s L =s R =s C =0,s O =-1,s M =1 is achieved. The trigger sample generation process is essentially an addition operation of pixel values, which can be represented by constraints on the addition gate. For the model's inference process for the trigger sample, constraints need to be constructed for different types of layers. For linear and convolutional layers consisting of addition and multiplication operations, these can be constructed using addition and multiplication gates with PLONKish constraints. For the validation of ReLU activation layers, an auxiliary selector s needs to be introduced. a ∈{0,1}, if the input x is greater than 0, then set s a =1, otherwise set s a =0. Then construct s a ·(ReLU(x)-x)=0 and (1-s a The constraints )·(ReLU(x)-0)=0 are used to verify the correctness of the activation process, where ReLU(x) represents the result of the input x after passing through the ReLU(·) activation function. All the above constraints constitute a zero-knowledge proof circuit C to be verified.

[0053] The zero-knowledge proof generation process: call the Prove(·) algorithm in the existing zero-knowledge proof protocol PlonK to prove the correctness of the zero-knowledge proof circuit C constructed using the PLONKish language, and generate a zero-knowledge proof π = Prove(C, w, t), where w represents the private input information, i.e. the pixel value of the trigger pattern. t represents the public input information, including the pixel value of the sample in the training set subset and the parameter information of the target model.

[0054] When modeling the detection model watermarking process, the PLONKish arithmetic circuit containing addition gates and multiplication gates is used to model the trigger sample generation process as a process of adding the corresponding pixel values of the trigger pattern and the clean training set picture, and the corresponding circuit is constructed using the addition gate.

[0055] For the circuit construction of the model forward inference process, it is divided into circuit construction for arithmetic layers and circuit construction for non-arithmetic layers. The former can be completed by combining addition gates and multiplication gates, such as linear layers, convolution layers, etc. in machine learning models, and the latter needs to be constructed through a lookup table, a bit decomposition operation, and then a circuit construction process for non-arithmetic layers such as ReLU layers.

[0056] Model ownership verification and arbitration step: the verification algorithm in the zero-knowledge proof protocol needs to be called to verify the correctness of the proof about the trigger sample generation process and the inference process of the model for the trigger sample. If the verification is passed, it means that the model ownership verification is passed.

[0057] In this embodiment, the step of detecting the model watermark and generating the proof can also use a publicly verifiable multi-party secure computation scheme instead of a zero-knowledge proof protocol. Specifically, a corresponding multi-party secure computation circuit is constructed for the two processes of making new trigger samples and the inference of the model for the trigger samples, and a publicly verifiable multi-party secure computation scheme is used to generate the correctness proof of the circuit operation.

[0058] More specifically, the correctness proof of the circuit operation generated by the multi-party secure computation scheme can be divided into two parts: the first is to construct a multi-party secure computation circuit, and the second is to call the Add(·) function and Mult(·) function in the existing multi-party secure computation protocol SPDZ to operate the circuit, where Add(·) represents addition operation and Mult(·) represents multiplication operation. Finally, the output of the circuit is obtained, and the correctness of the result is verified by the MAC code in SPDZ.

[0059] Constructing a multi-party secure computation circuit: similar to constructing a zero-knowledge proof circuit, the trigger sample generation process is represented by addition, and the model inference process is represented by addition and multiplication.

[0060] The application further provides a machine learning image classification model ownership verification system, which can be implemented by executing the process steps of the machine learning image classification model ownership verification method, i.e., the machine learning image classification model ownership verification method can be understood by those skilled in the art as a preferred embodiment of the machine learning image classification model ownership verification system.

[0061] Those skilled in the art know that, in addition to implementing the system provided by the application and each device, module and unit thereof in the form of pure computer readable program code, the system provided by the application and each device, module and unit thereof can also be implemented in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers and embedded microcontrollers, etc. by logically programming the method steps to achieve the same functions. Therefore, the system provided by the application and each device, module and unit thereof can be considered as a hardware component, and the devices, modules and units included therein for achieving various functions can also be considered as structures within the hardware component; the devices, modules and units for achieving various functions can also be considered as both software modules implementing methods and structures within the hardware component.

[0062] The specific embodiments of the application are described above. It needs to be understood that the application is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essential content of the application. The embodiments of the present application and the features in the embodiments can be arbitrarily combined with each other without conflict.

Claims

1. A method for verifying ownership of a machine learning image classification model, characterized in that, include: Step S1: Generate a set of trigger samples; Step S2: Use backdoor training technology to fine-tune the image classification model so that the fine-tuned image classification model can produce incorrect prediction results that meet the preset requirements for the trigger sample set while maintaining normal prediction function; Step S3: First, overlay the pre-selected trigger pattern onto a clean training set image to create a new trigger sample. Then, use this trigger sample as the model input and use the target model to complete the forward reasoning process to obtain the model's prediction result. Finally, model the trigger sample generation process and the model's reasoning process for the trigger sample as a zero-knowledge proof circuit. Use the proof algorithm in the zero-knowledge proof protocol to generate the correctness proof of the above two processes without exposing the trigger pattern, and submit the correctness proof to a third-party organization. Step S4: Call the verification algorithm in the zero-knowledge proof protocol to verify the correctness of the proof regarding the trigger sample generation process and the model's reasoning process for the trigger sample. If the verification passes, it means that the model ownership verification has passed. Step S3 includes: invoking the zero-knowledge proof protocol PlonK. Algorithm for zero-knowledge proof circuits constructed using PLONKish language Prove the correctness and generate a zero-knowledge proof. ;in, Represents the pixel value of the trigger pattern; This represents the pixel values ​​of samples in the training subset and the parameter information of the fine-tuned image classification model.

2. The ownership verification method for machine learning image classification models according to claim 1, characterized in that, Step S1 includes: overlaying the trigger pattern onto a subset of the training set of the image classification model to obtain a set of trigger samples; The training set subset of the image classification model includes: setting a subset size ratio, and randomly selecting samples from each category of the training set as a training set subset according to the set subset size ratio; The step of overlaying the trigger pattern onto a subset of the training set of the image classification model includes: representing each pixel in the two images as an RGB three-channel form, and then adding the values ​​of the three channels at the corresponding positions as the overlay result. If the result is greater than the threshold of 255, the overlay result is set to 255.

3. An ownership verification system for a machine learning image classification model, characterized in that, include: Module M1: Generates a set of trigger samples; Module M2: Uses backdoor training technology to fine-tune the image classification model so that the fine-tuned image classification model can produce incorrect prediction results that meet preset requirements for the trigger sample set while maintaining normal prediction function; Module M3: First, the pre-selected trigger pattern is superimposed on a clean training set image to create a new trigger sample. Then, the trigger sample is used as the model input, and the target model is used to complete the forward reasoning process to obtain the model's prediction result. Finally, the trigger sample generation process and the model's reasoning process for the trigger sample are modeled as a zero-knowledge proof circuit. Using the proof algorithm in the zero-knowledge proof protocol, the correctness proof of the above two processes is generated without exposing the trigger pattern, and the correctness proof is submitted to a third-party organization. Module M4: Calls the verification algorithm in the zero-knowledge proof protocol to verify the correctness of the proof regarding the trigger sample generation process and the model's reasoning process for the trigger sample. If the verification passes, it means that the model ownership verification has passed. Module M3 includes: calling the zero-knowledge proof protocol PlonK. Algorithm for zero-knowledge proof circuits constructed using PLONKish language Prove the correctness and generate a zero-knowledge proof. ;in, Represents the pixel value of the trigger pattern; This represents the pixel values ​​of samples in the training subset and the parameter information of the fine-tuned image classification model.

4. The ownership verification system for machine learning image classification models according to claim 3, characterized in that, The module M1 includes: overlaying the trigger pattern onto a subset of the training set of the image classification model to obtain a set of trigger samples; The training set subset of the image classification model includes: setting a subset size ratio, and randomly selecting samples from each category of the training set as a training set subset according to the set subset size ratio; The step of overlaying the trigger pattern onto a subset of the training set of the image classification model includes: representing each pixel in the two images as an RGB three-channel form, and then adding the values ​​of the three channels at the corresponding positions as the overlay result. If the result is greater than the threshold of 255, the overlay result is set to 255.

Images