Unlearnable image privacy protection method based on frequency domain

Abstract

The invention discloses an unlearnable image privacy protection method based on a frequency domain. The unlearnable image privacy protection method comprises the steps of performing color space conversion on an original RGB image needing to be protected; for each channel, converting the channel from a spatial domain to a frequency domain by using discrete cosine transform to obtain a DCT coefficient table of each channel; initializing a WGAN generator for each channel and inputting a corresponding DCT coefficient table to obtain an unlearnable disturbance channel DCT coefficient table, adding the unlearnable disturbance channel DCT coefficient table to a DCT coefficient table of a corresponding channel of an original image, combining the channels after inverse discrete cosine transform, and performing color space conversion to obtain an unlearnable image; the method comprises the following steps: initializing a WGAN discriminator and a classifier, inputting an unlearnable image, respectively calculating training loss, feeding back and updating the discriminator and the classifier, obtaining generator loss by using the updated discriminator and classifier, updating the generator, and carrying out loop iteration training for a certain round to obtain a disturbance generator; and generating unlearnable disturbance for the image to obtain an unlearnable image.

Description

Technical Field

[0001] The invention relates to a frequency domain-based non-learnable image privacy protection method, belonging to the technical field of image processing. Background Art

[0002] The rapid development of deep neural networks (DNNs) has made them have great development prospects in all walks of life. A well-trained DNN model depends largely on the number of training samples, but there are no shortage of malicious attackers who abuse data to train malicious models or unauthorized third parties who use data to train commercial models, which has aroused public concern about data privacy issues. Especially in the field of biomedicine, this has also caused patients and biomedical institutions to worry about data privacy and resist sharing data, which has greatly hindered the progress of the entire biomedical research field. Therefore, from the perspective of data protection, it is crucial to propose methods to solve privacy violations and data abuse to prevent unauthorized model training, while also allowing other normal purposes such as auxiliary diagnosis, medical records, etc. The current protection of image data privacy is mainly based on data poisoning methods, which usually add perturbations that are invisible to the naked eye to the image. The defender poisons the image and publishes the poisoned image, and the attacker uses the poisoned image downloaded from the Internet to train the model, which usually cannot perform well on normal data sets.

[0003] Huang et al. first studied unlearnable images. Inspired by the intuition that the larger the loss function is during model training, the more information the model can learn from it, they proposed unlearnable images, that is, generating perturbations that can minimize the DNN loss function and adding them to clean images. DNN believes that it has no information to learn, that is, the image becomes an unlearnable image. Ma et al. found that simply minimizing the loss function to generate unlearnable images failed after adding adversarial attacks to the model, so they proposed a method for generating unlearnable images that is resistant to adversarial attacks: robust unlearnable images. It adds adversarial training noise to the training process of the loss function minimization scheme, and adds transformation functions to the image to enhance robustness.

[0004] However, the above method has the following disadvantages: (1) Experiments have found that the non-learnable images obtained by the current method of minimizing the training loss function are not effective for medical images and are almost ineffective on some medical data sets. (2) Perturbations based on the spatial domain will affect the image visualization effect to a certain extent. Generally speaking, the larger the perturbation radius, the more robust the perturbation is, but it is also easier for the human eye to detect, affecting the normal use of the image. In addition, there are currently many defense methods for poisoning methods based on the spatial domain, which limits their use. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a frequency-domain-based non-learnable image privacy protection method, which adds non-learnable disturbances in the frequency domain to achieve the effect of protecting image privacy.

[0006] The present invention adopts the following technical solutions to solve the above technical problems:

[0007] A frequency domain-based non-learnable image privacy protection method comprises the following steps:

[0008] Step 1: for the original image to be protected, convert it from RGB color space to YUV color space to obtain a YUV color space image;

[0009] Step 2, for the Y, U and V channels of the YUV color space image, use discrete cosine transform to convert each channel from the spatial domain to the frequency domain to obtain a DCT coefficient table for each channel;

[0010] Step 3: Initialize a WGAN generator for each channel of Y, U, and V and Initialize a WGAN discriminator D according to the original image to be protected 1 and a classifier C 1 ; The WGAN discriminator and classifier are updated simultaneously, and the WGAN generator is updated once every m times the WGAN discriminator and classifier are updated simultaneously, m ≥ 1;

[0011] Step 4: For the i-th iteration of the WGAN discriminator and classifier, input the corresponding DCT coefficient table into the current WGAN generator of each channel to obtain the perturbed DCT coefficient table of each channel and Add each perturbation DCT coefficient table to the DCT coefficient table of the corresponding channel to obtain a new DCT coefficient table for each channel and

[0012] Step 5: New DCT coefficient table for each channel and Perform inverse discrete cosine transform, convert each channel from frequency domain back to spatial domain, and merge channels to obtain a new YUV color space image. Convert the new YUV color space image back to RGB color space to obtain the non-learnable image of the i-th iteration.

[0013] Step 6: The non-learnable image of the i-th iteration Input to WGAN discriminator D i-1 and classifier C i-1 , respectively calculate the WGAN discriminator D i-1 and classifier C i-1The loss function and use and Update the WGAN discriminator D respectively i-1 and classifier C i-1 , get the WGAN discriminator D i and classifier C i ;

[0014] Step 7: For the jth iteration of the WGAN generator, and Input the corresponding DCT coefficient table, and follow the same operation as steps 4-5 to obtain the non-learnable image of the jth iteration Will Input to the current WGAN discriminator and the current classifier, calculate the loss function of the current WGAN discriminator and classifier respectively, use the loss function of the current WGAN discriminator and classifier to calculate the loss function of the WGAN generator, and use the optimizer to update the WGAN generator and Get the WGAN generator and

[0015] Step 8: Determine whether the WGAN generator has reached the preset number of iterations. If so, replace the WGAN generator of the last iteration with and as the trained perturbation generator; otherwise, let i=i+1 and return to step 4;

[0016] Step 9: Use the trained perturbation generator to generate a non-learnable image for the image to be protected by following the same operations as steps 4 and 5.

[0017] As a preferred solution of the present invention, the WGAN discriminator D i , classifier C i And the WGAN generator and Both are multi-layer convolutional neural networks, and the classifier C i And the WGAN generator and Both include residual modules.

[0018] As a preferred solution of the present invention, in step 2, the formula of discrete cosine transform is as follows:

[0019]

[0020] in, Represents the value of the matrix obtained by discrete cosine transform at the coordinates [u,v], N represents the image size, and x[k,m] represents the pixel value of the image at coordinate [k,m].

[0021] As a preferred solution of the present invention, in step 5, the formula of inverse discrete cosine transform is as follows:

[0022]

[0023] in, Represents the pixel value of the matrix obtained by inverse discrete cosine transform at coordinates [k,m].

[0024] As a preferred solution of the present invention, in step 6, the loss function of the WGAN discriminator is:

[0025]

[0026] Among them, l D represents the WGAN discriminator loss, Represents the actual data distribution The expectation of a sample x is is the probability distribution of the real data, x is a random sample drawn from this distribution, is the judgment result of the discriminator on sample x, Represents the non-learnable image x generated by the generator ue expectations, is the probability distribution of non-learnable images, is the discriminator for the unlearnable image x ue The judgment result, λ represents the penalty coefficient, Indicates interpolation non-learnable samples expectations, is the interpolated distribution between the real data and the data generated by the generator, is the interpolated non-learnable image after adding Gaussian noise, Indicates about Find the gradient, Represents the discriminator for unlearnable images The determination result;

[0027] The loss function of the classifier is:

[0028]

[0029] Among them, l C represents the classifier loss, y represents the true label, represents the label predicted by the classifier;

[0030] The loss function of the WGAN generator is:

[0031]

[0032] Among them, l G represents the WGAN generator, α and β are the weights of the two parts of the loss respectively.

[0033] As a preferred solution of the present invention, in step 7, the optimizer is Adam.

[0034] A computer device comprises a memory, a processor, and a computer program stored in the memory and capable of running on the processor. When the processor executes the computer program, the steps of the frequency domain-based non-learnable image privacy protection method are implemented.

[0035] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the frequency-domain-based non-learnable image privacy protection method are implemented.

[0036] Compared with the prior art, the present invention adopts the above technical solution and has the following technical effects:

[0037] 1. This invention supports the generation of non-learnable images under gray-box conditions by introducing the idea of ​​WGAN, that is, there is no need to obtain the internal architecture and parameters of the malicious model, but only the output of the malicious model. After the generator is trained, any image can be perturbed and added without interacting with the model. At the same time, the adopted WGAN solves the problem of unstable training of the original GAN, and no longer needs to carefully balance the training degree of the generator and the discriminator; it basically solves the problem of gradient explosion and ensures the diversity of generated samples.

[0038] 2. For medical images with monotonous background colors, relatively prominent lesion areas, and higher concealment requirements, the present invention uses frequency-domain-based perturbations to reduce the visibility of perturbations and improve concealment. At the same time, there are currently few frequency-domain-based poisoning methods, and their corresponding countermeasure methods are relatively scarce. The frequency-domain-based non-learnable perturbations have higher robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a flow chart of a frequency domain-based non-learnable image privacy protection method of the present invention;

[0040] Figure 2 is a schematic diagram of a frequency domain-based non-learnable image privacy protection method of the present invention;

[0041] Figure 3 It is a schematic diagram of a simulation protection system based on the method of the present invention. DETAILED DESCRIPTION

[0042] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention.

[0043] like Figure 1 and Figure 2 As shown, the present invention proposes a frequency domain-based non-learnable image privacy protection method, comprising the following steps:

[0044] Step 1: Color space conversion

[0045] 1.1 Data protection party from the clean data set D clean Select the original RGB image x that needs to be protected;

[0046] 1.2 The data protector converts x from RGB color space to YUV color space to obtain image x ′ .

[0047] The color space conversion here includes RGB to YUV, YUV back to RGB, RGB to YCbCr, YCbCr back to RGB color space, etc.

[0048] Step 2: Frequency Domain Conversion

[0049] 2.1 For image x ′ For each channel of Y, U, and V, the data protection party uses discrete cosine transform (DCT) to convert the image from the spatial domain to the frequency domain, and obtains the DCT coefficient table D for each channel respectively. Y , D U , D V .

[0050] The discrete cosine transform (DCT) formula is:

[0051]

[0052] in, N represents the image size. The above discrete cosine transform (DCT) formula uses the value of image x at coordinates [k,m] to calculate The u and v items of .

[0053] Step 3: Unlearnable Image Generation

[0054] 3.1 According to DCT coefficient table D Y , D U , D V , initialize a WGAN for each channel to generate and

[0055] 3.2 Generator for each channel and Input the corresponding DCT coefficient table to obtain the perturbed DCT coefficient table of the three channels of x in the frequency domain and

[0056] 3.3 Add to the DCT coefficient table of the corresponding channel to obtain the non-learnable image The three channel DCT coefficient table and

[0057] 3.4 Using inverse discrete cosine transform (IDCT) to transform the non-learnable image The three channel DCT coefficient table and Convert back to the spatial domain to get the non-learnable image Y, U, V color space image channels;

[0058] The inverse discrete cosine transform (IDCT) formula is:

[0059]

[0060] 3.5 Finally, merge the channels and convert the color space back to RGB color space to obtain a non-learnable image

[0061] Step 4: Perturbation Generator Training

[0062] 4.1 Initialize a WGAN discriminator D based on the original RGB image x 1 and a classifier C (a malicious model simulating an attacker) 1 ;

[0063] 4.2 Obtaining unlearnable images through the current generator and conversion process Used to train the discriminator D i-1 and classifier C i-1 , respectively calculate the training and And feedback updates the discriminator and classifier;

[0064] 4.3 The WGAN discriminator and classifier are updated m times at the same time, and the WGAN generator is updated once, m ≥ 1. For the jth iteration of the WGAN generator, the WGAN generator of each channel and Input the corresponding DCT coefficient table and follow the same operation as above to get the non-learnable image of the jth iteration Will Input to the current WGAN discriminator and the current classifier, calculate the loss function of the current WGAN discriminator and classifier respectively, use the loss function of the current WGAN discriminator and classifier to calculate the loss function of the WGAN generator, and use the optimizer to update the WGAN generator and Get the WGAN generator and In the above process, the discriminator D, the classifier C and the generator G Y , G U , G V Three-party game, cyclic iterative training and updating the generator for a certain number of rounds to obtain the perturbation generator

[0065] The loss function of the discriminator is:

[0066]

[0067] in, Represents the actual data distribution The expectation of a sample x. Here is the probability distribution of the real data, and x is a random sample drawn from this distribution. For the discriminator The judgment result of the original image x. Represents the non-learnable image x generated by the generator ue expectations, is the probability distribution of non-learnable images, It is a discriminator For non-learnable images x ue The decision result. λ represents the penalty coefficient, which is the gradient constraint, which is used to require that the L2 norm of the gradient relative to the original input be constrained to be around 1. is the interpolated distribution between the real data and the data generated by the generator, is the non-learnable image x ue Obtained by adding Gaussian noise.

[0068] The loss function of classifier C is the cross entropy loss function:

[0069]

[0070] Among them, y represents the true label, Represents the label predicted by the classifier.

[0071] The loss function of the generator is:

[0072]

[0073] Among them, α and β are the weights of the two parts of loss respectively.

[0074] Discriminator D, generator G Y , G U , G V and classifier C are all multi-layer convolutional neural networks, where the generator G Y , G U , G V And the classifier C contains a residual module.

[0075] Step 5: Add perturbations to the data to get unlearnable images

[0076] The data protection party uses the trained perturbation generator Generate unlearnable perturbations for the data to be protected to obtain unlearnable images x ue .

[0077] like Figure 3 As shown in Figure 1, a simulated unlearnable image privacy protection system consists of three parts: data protection party, social platform and unauthorized data users. The data protection party adds unlearnable perturbations to the image to obtain the unlearnable image and publishes it to the social platform through the network or other sharing channels. The unauthorized data user downloads the unlearnable image from the social platform through crawlers or other channels and trains the malicious model C. m , the model cannot learn or learns less image information to achieve the purpose of protecting data. Malicious Model C m It is a multi-layer convolutional neural network with a residual module.

[0078] Since neural networks usually prefer to learn simple features, i.e. shortcuts, non-learnable perturbations provide the model with easy-to-learn features (shortcuts), so the model cannot learn or learns less complex high-dimensional feature information, thereby achieving the effect of protecting the user's medical image privacy.

[0079] Based on the same inventive concept, an embodiment of the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the aforementioned frequency domain-based non-learnable image privacy protection method are implemented.

[0080] Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of the aforementioned frequency-domain-based non-learnable image privacy protection method are implemented.

[0081] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0082] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0083] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0084] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0085] The above embodiments are only for illustrating the technical idea of ​​the present invention, and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the present invention.

Claims

1. A frequency-domain-based non-learnable image privacy protection method, characterized in that: The steps include: Step 1: for the original image to be protected, convert it from RGB color space to YUV color space to obtain a YUV color space image; Step 2, for the Y, U and V channels of the YUV color space image, use discrete cosine transform to convert each channel from the spatial domain to the frequency domain to obtain a DCT coefficient table for each channel; Step 3: Initialize a WGAN generator for each channel of Y, U, and V and Initialize a WGAN discriminator D according to the original image to be protected 1 and a classifier C 1 ; The WGAN discriminator and classifier are updated simultaneously, and the WGAN generator is updated once every m times the WGAN discriminator and classifier are updated simultaneously, m ≥ 1; Step 4: For the i-th iteration of the WGAN discriminator and classifier, input the corresponding DCT coefficient table into the current WGAN generator of each channel to obtain the perturbed DCT coefficient table of each channel and Add each perturbation DCT coefficient table to the DCT coefficient table of the corresponding channel to obtain a new DCT coefficient table for each channel and Step 5: New DCT coefficient table for each channel and Perform inverse discrete cosine transform, convert each channel from frequency domain back to spatial domain, and merge channels to obtain a new YUV color space image. Convert the new YUV color space image back to RGB color space to obtain the non-learnable image of the i-th iteration. Step 6: The non-learnable image of the i-th iteration Input to WGAN discriminator D i-1 and classifier C i-1 , respectively calculate the WGAN discriminator D i-1 and classifier C i-1 The loss function and use and Update the WGAN discriminator D respectively i-1 and classifier C i-1 , get the WGAN discriminator D i and classifier C i ; Step 7: For the jth iteration of the WGAN generator, and Input the corresponding DCT coefficient table, and follow the same operation as steps 4-5 to obtain the non-learnable image of the jth iteration Will Input to the current WGAN discriminator and the current classifier, calculate the loss function of the current WGAN discriminator and classifier respectively, use the loss function of the current WGAN discriminator and classifier to calculate the loss function of the WGAN generator, and use the optimizer to update the WGAN generator and Get the WGAN generator and Step 8: Determine whether the WGAN generator has reached the preset number of iterations. If so, replace the WGAN generator of the last iteration with and as the trained perturbation generator; otherwise, let i=i+1 and return to step 4; Step 9: Use the trained perturbation generator to generate a non-learnable image for the image to be protected by following the same operations as steps 4 and 5.

2. The frequency-domain-based non-learnable image privacy protection method according to claim 1, characterized in that: The WGAN discriminator D i , Classifier C i And the WGAN generator and Both are multi-layer convolutional neural networks, and the classifier C i And the WGAN generator and Both include residual modules.

3. The frequency-domain-based non-learnable image privacy protection method according to claim 1, characterized in that: In step 2, the formula of discrete cosine transform is as follows: in, Represents the value of the matrix obtained by discrete cosine transform at the coordinates [u,v], N represents the image size, and x[k,m] represents the pixel value of the image at coordinate [k,m].

4. The frequency-domain-based non-learnable image privacy protection method according to claim 3, characterized in that: In step 5, the formula for inverse discrete cosine transform is as follows: in, Represents the pixel value of the matrix obtained by inverse discrete cosine transform at coordinates [k,m].

5. The frequency-domain-based non-learnable image privacy protection method according to claim 1, characterized in that: In step 6, the loss function of the WGAN discriminator is: Among them, l D represents the WGAN discriminator loss, Represents the actual data distribution The expectation of a sample x is is the probability distribution of the real data, x is a random sample drawn from this distribution, is the judgment result of the discriminator on sample x, Represents the non-learnable image x generated by the generator ue expectations, is the probability distribution of non-learnable images, is the discriminator for the unlearnable image x ue The judgment result, λ represents the penalty coefficient, Indicates interpolation non-learnable samples expectations, is the interpolated distribution between the real data and the data generated by the generator, is the interpolated non-learnable image after adding Gaussian noise, Indicates about Find the gradient, Represents the discriminator for unlearnable images The determination result; The loss function of the classifier is: Among them, l C represents the classifier loss, y represents the true label, represents the label predicted by the classifier; The loss function of the WGAN generator is: Among them, l G represents the WGAN generator, α and β are the weights of the two parts of the loss respectively.

6. The frequency-domain-based non-learnable image privacy protection method according to claim 1, characterized in that: In step 7, the optimizer is Adam.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that: When the processor executes the computer program, the steps of the frequency domain-based non-learnable image privacy protection method according to any one of claims 1 to 6 are implemented.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the frequency domain-based non-learnable image privacy protection method according to any one of claims 1 to 6 are implemented.

Images