A frequency-domain-based non-learnable image privacy protection method

Abstract

The present invention discloses a non-learnable image privacy protection method based on the frequency domain, including: performing color space conversion on the original RGB image to be protected; for each channel, using the discrete cosine transform to convert it from the spatial domain to the frequency domain to obtain the DCT coefficient table of each channel; initializing the WGAN generator for each channel and inputting the corresponding DCT coefficient table, obtaining the DCT coefficient table of the non-learnable perturbation for each channel and adding it to the DCT coefficient table of the corresponding channel of the original image, merging the channels after the inverse discrete cosine transform, and then obtaining the non-learnable image through color space conversion; initializing the WGAN discriminator and classifier, inputting the non-learnable image, calculating the training loss respectively and feeding back to update the discriminator and classifier, obtaining the generator loss using the updated discriminator and classifier, updating the generator, and obtaining the perturbation generator after training for a certain number of rounds through cyclic iteration; that is, generating a non-learnable perturbation for the image to obtain a non-learnable image.

Description

Technical Field

[0001] The present invention relates to a frequency-domain-based unlearnable image privacy protection method, belonging to the technical field of image processing. Background Art

[0002] The rapid development of deep neural networks (DNNs) has made them have great development prospects in all walks of life. A well-trained DNN model largely depends on the number of training samples. However, there are malicious attackers who abuse data to train malicious models or unauthorized third parties apply data to train commercial models, which has attracted public attention to data privacy issues. Especially in the biomedical field, this has also caused concerns about data privacy among patients and biomedical institutions, resisting data sharing, and greatly hindering the progress of the entire biomedical research field. Therefore, from the perspective of data protection parties, it is crucial to propose methods to solve privacy violations and data abuse to avoid unauthorized model training while also allowing other normal uses such as auxiliary diagnosis, medical records, etc. Currently, the protection of image data privacy is mainly based on data poisoning methods, and data poisoning usually adds invisible perturbations to images for the naked eye. The defender poisons the image and makes the poisoned image public. The attacker uses the poisoned image downloaded from the network to train the model, and usually this model cannot perform well on the normal data set.

[0003] Huang et al. first studied unlearnable images. Inspired by the intuition that the larger the loss function in the model training process, the more information the data model can learn, they proposed unlearnable images, that is, generating perturbations that can minimize the DNN loss function and adding them to clean images. The DNN believes that there is no information to learn, that is, the image becomes an unlearnable image. Ma et al. found that simply minimizing the loss function to generate unlearnable images fails after the model is added with adversarial attacks, so they proposed a method for generating unlearnable images that resists adversarial attacks: robust unlearnable images. During the training process of the minimum loss function scheme, adversarial training noise is added, and at the same time, a transformation function is added to the image to enhance robustness.

[0004] However, the above methods have the following disadvantages: (1) Through experiments, it is found that the unlearnable images obtained by the current method of minimizing the training loss function have poor effects on medical images and are almost ineffective on some medical data sets. (2) Perturbations based on the spatial domain will affect the image visualization effect to a certain extent. Generally speaking, when the perturbation radius is larger, the perturbation is more robust but also more easily detectable by the human eye, affecting the normal use of the image; and there are already many defense methods for the current poisoning methods based on the spatial domain, and their use is limited. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a frequency-domain-based unlearnable image privacy protection method, which adds unlearnable perturbations in the frequency domain to achieve the effect of protecting image privacy.

[0006] The present invention adopts the following technical solutions to solve the above technical problems:

[0007] A frequency-domain-based unlearnable image privacy protection method includes the following steps:

[0008] Step 1, for the original image to be protected, convert it from the RGB color space to the YUV color space to obtain a YUV color space image;

[0009] Step 2, for the Y, U, and V channels of the YUV color space image, use the discrete cosine transform to convert each channel from the spatial domain to the frequency domain to obtain the DCT coefficient table of each channel;

[0010] Step 3, initialize a WGAN generator for each of the Y, U, and V channels and Initialize a WGAN discriminator D according to the original image to be protected 1 and a classifier C 1 ; The WGAN discriminator and the classifier are updated simultaneously, and the WGAN generator is updated once every m times when the WGAN discriminator and the classifier are updated simultaneously, where m≥1;

[0011] Step 4, for the i-th iteration of the WGAN discriminator and the classifier, input the corresponding DCT coefficient table into the current WGAN generator of each channel to obtain the perturbed DCT coefficient table of each channel and Add the perturbed DCT coefficient tables to the DCT coefficient tables of the corresponding channels to obtain the new DCT coefficient tables of each channel and

[0012] Step 5, for the new DCT coefficient tables of each channel and Perform the inverse discrete cosine transform to convert each channel back from the frequency domain to the spatial domain, and perform channel merging to obtain a new YUV color space image, and convert the new YUV color space image back to the RGB color space to obtain the unlearnable image of the i-th iteration

[0013] Step 6, input the unlearnable image of the i-th iteration into the WGAN discriminator D i-1 and the classifier C i-1 , and calculate the WGAN discriminator D i-1 and the classifier C i-1Loss function and Use and to update the WGAN discriminator D i-1 and the classifier C i-1 respectively, and obtain the WGAN discriminator D i and the classifier C i ;

[0014] Step 7, for the j-th iteration of the WGAN generator, input the corresponding DCT coefficient table into the WGAN generator and in each channel, and perform the same operations as in Steps 4 - 5 to obtain the non-learnable image of the j-th iteration Input into the current WGAN discriminator and the current classifier, calculate the loss functions of the current WGAN discriminator and the classifier respectively, calculate the loss function of the WGAN generator using the loss functions of the current WGAN discriminator and the classifier, and update the WGAN generator using the optimizer and to obtain the WGAN generator and

[0015] Step 8, determine whether the WGAN generator has reached the preset number of iterations. If so, use the WGAN generator and of the last iteration as the trained perturbation generator; otherwise, set i = i + 1 and return to Step 4;

[0016] Step 9, use the trained perturbation generator to generate a non-learnable image for the image to be protected according to the same operations as in Steps 4 - 5.

[0017] As a preferred solution of the present invention, the WGAN discriminator D i , the classifier C i , and the WGAN generator and are all multi-layer convolutional neural networks, and the classifier C i , and the WGAN generator and all include residual modules.

[0018] As a preferred solution of the present invention, in Step 2, the formula for discrete cosine transform is as follows:

[0019]

[0020] where, Denotes the value of the matrix obtained by discrete cosine transform at coordinates [u, v]. N represents the image size, and x[k, m] represents the pixel value of the image at coordinates [k, m].

[0021] As a preferred embodiment of the present invention, in step 5, the formula for inverse discrete cosine transform is as follows:

[0022]

[0023] Wherein, Denotes the pixel value of the matrix obtained by inverse discrete cosine transform at coordinates [k, m].

[0024] As a preferred embodiment of the present invention, in step 6, the loss function of the WGAN discriminator is:

[0025]

[0026] Wherein, l D Denotes the WGAN discriminator loss, Denotes the expectation of the sample x of the true data distribution of, is the probability distribution of the true data, x is a sample randomly drawn from this distribution, is the determination result of the discriminator for the sample x, Denotes the unlearnable image x generated by the generator ue of expectation, is the probability distribution of the unlearnable image, is the determination result of the discriminator for the unlearnable image x ue of, λ represents the penalty coefficient, Denotes the interpolated unlearnable sample of expectation, is the interpolation distribution between the true data and the data generated by the generator, is the interpolated unlearnable image after adding Gaussian noise, Denotes with respect to take the gradient, Denotes the determination result of the discriminator for the unlearnable image of;

[0027] The loss function of the classifier is:

[0028]

[0029] Wherein, l C Denotes the classifier loss, y represents the true label, Denotes the label predicted by the classifier;

[0030] The loss function of the WGAN generator is as follows:

[0031]

[0032] Among them, l G represents the WGAN generator, and α and β are the weights of the two parts of the loss respectively.

[0033] As a preferred solution of the present invention, in step 7, the optimizer is Adam.

[0034] A computer device includes a memory, a processor, and a computer program stored in the memory and capable of running on the processor. When the processor executes the computer program, the steps of the frequency-domain-based unlearnable image privacy protection method are implemented.

[0035] A computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the steps of the frequency-domain-based unlearnable image privacy protection method are implemented.

[0036] Compared with the prior art, the present invention adopts the above technical solutions and has the following technical effects:

[0037] 1. By introducing the idea of WGAN, the present invention supports the generation of unlearnable images under the gray-box condition, that is, without obtaining information such as the internal architecture and parameters of the malicious model, only by obtaining the output of the malicious model. After training the generator, perturbations can be added to any image without interacting with the model. At the same time, the adopted WGAN solves the problem of unstable training of the original GAN, and no longer needs to carefully balance the training degree of the generator and the discriminator; basically solves the problem of gradient explosion, ensuring the diversity of the generated samples.

[0038] 2. For medical images with monotonous background colors, relatively prominent lesion areas, and higher concealment requirements, the present invention uses frequency-domain-based perturbations to reduce the visibility of the perturbations and improve the concealment. At the same time, there are relatively few frequency-domain-based poisoning methods, and the corresponding countermeasures are relatively scarce. The frequency-domain-based unlearnable perturbations have high robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 is a flowchart of a frequency-domain-based unlearnable image privacy protection method of the present invention;

[0040] Figure 2 is a schematic diagram of a frequency-domain-based unlearnable image privacy protection method of the present invention;

[0041] Figure 3 is a schematic diagram of a simulation protection system based on the method of the present invention. Detailed implementation manners

[0042] The following is a detailed description of the implementation manners of the present invention. Examples of the implementation manners are shown in the accompanying drawings. The implementation manners described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0043] As Figure 1 and Figure 2 shown, the present invention proposes a frequency-domain-based unlearnable image privacy protection method, including the following steps:

[0044] Step 1: Color space conversion

[0045] 1.1 The data protection party selects the original RGB image x to be protected from the clean data set D clean .

[0046] 1.2 The data protection party converts x from the RGB color space to the YUV color space to obtain the image x ′ .

[0047] The color space conversion here includes RGB to YUV, YUV back to RGB, RGB to YCbCr, and YCbCr back to RGB color spaces, etc.

[0048] Step 2: Frequency domain conversion

[0049] 2.1 For each of the Y, U, and V channels of the image x ′ , the data protection party uses the discrete cosine transform (DCT) to convert the image from the spatial domain to the frequency domain, and obtains the DCT coefficient tables D Y , D U , and D V of each channel respectively.

[0050] The discrete cosine transform (DCT) formula is:

[0051]

[0052] Wherein, N represents the image size, and the above discrete cosine transform (DCT) formula calculates the u-th and v-th terms of using the value of the image x at the coordinates [k, m].

[0053] Step 3: Generation of unlearnable images

[0054] 3.1 According to the DCT coefficient tables D Y , D U , and D V , initialize a WGAN generator for each channel and

[0055] 3.2 Generator for each channel and Input the corresponding DCT coefficient table to obtain the perturbation DCT coefficient table of the three channels of x in the frequency domain and

[0056] 3.3 Add to the DCT coefficient table of the corresponding channel to obtain the non-learnable image Three channel DCT coefficient tables and

[0057] 3.4 Using Inverse Discrete Cosine Transform (IDCT) to transform the non-learnable image Three channel DCT coefficient tables and Convert back to the spatial domain to obtain a non-learnable image Y, U, V color space image channels;

[0058] The inverse discrete cosine transform (IDCT) formula is:

[0059]

[0060] 3.5 Finally, merge the channels and convert the color space back to RGB color space to obtain the non-learnable image

[0061] Step 4: Perturbation Generator Training

[0062] 4.1 Initialize a WGAN discriminator D based on the original RGB image x 1 and a classifier C (a malicious model simulating an attacker) 1 ;

[0063] 4.2 Obtaining non-learnable images through the current generator and conversion process Used to train the discriminator D i-1 and classifier C i-1 , respectively calculate the training and And feedback updates the discriminator and classifier;

[0064] 4.3 The WGAN discriminator and classifier are updated m times simultaneously, and the WGAN generator is updated once, where m ≥ 1. For the jth iteration of the WGAN generator, the WGAN generator of each channel and Input the corresponding DCT coefficient table and follow the same operation as above to get the non-learnable image of the jth iteration. Will Input into the current WGAN discriminator and the current classifier, calculate the loss functions of the current WGAN discriminator and the classifier respectively, calculate the loss function of the WGAN generator using the loss functions of the current WGAN discriminator and the classifier, and update the WGAN generator using the optimizer and Obtain the WGAN generator and In the above process, the discriminator D, the classifier C, and the generator G Y 、G U 、G V The three-party game, after cyclic iterative training and updating the generator for a certain number of rounds, obtain the perturbed generator

[0065] The loss function of the discriminator is:

[0066]

[0067] Among them, represents the expectation of the sample x of the true data distribution Here, is the probability distribution of the true data, and x is a sample randomly drawn from this distribution, is the judgment result of the discriminator on the original image x. Similarly represents the expectation of the unlearnable image x generated by the generator ue , is the probability distribution of the unlearnable image, is the discriminator 's judgment result on the unlearnable image x ue 。λ represents the penalty coefficient, and this part is the gradient constraint, which is used to require that the L2 norm of the gradient with respect to the original input be constrained around 1. is the interpolation distribution between the true data and the data generated by the generator, is obtained by adding Gaussian noise to the unlearnable image x ue 。

[0068] The loss function of the classifier C is the cross-entropy loss function:

[0069]

[0070] Among them, y represents the true label, represents the label predicted by the classifier.

[0071] The loss function of the generator is:

[0072]

[0073] Among them, α and β are the weights of the two parts of the loss respectively.

[0074] The discriminator D and the generator G Y , G U , G V And the classifier C are all multi-layer convolutional neural networks. Among them, the generator G Y , G U , G V And the classifier C contain residual modules.

[0075] Step 5: Add perturbations to the data to obtain unlearnable images

[0076] The data protector uses the trained perturbation generator To generate unlearnable perturbations for the data to be protected to obtain the unlearnable image x ue .

[0077] As Figure 3 Shown, a simulated unlearnable image privacy protection system consists of three parts: a data protector, a social platform, and an unauthorized data user. The data protector adds unlearnable perturbations to the image to obtain an unlearnable image and publicly releases it to the social platform through the network or other sharing channels. The unauthorized data user downloads the unlearnable image from the social platform through web crawlers or other channels and trains a malicious model C m , The model cannot learn or learn less image information to achieve the purpose of protecting data. The malicious model C m Is a multi-layer convolutional neural network and contains residual modules.

[0078] Since neural networks usually prefer to learn simple features, that is, shortcuts, the unlearnable perturbations provide features (shortcuts) that are easy to learn for the model. Therefore, the model cannot learn or learn less complex high-dimensional feature information, thus achieving the effect of protecting the privacy of users' medical images.

[0079] Based on the same inventive concept, an embodiment of the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the foregoing frequency-domain-based unlearnable image privacy protection method.

[0080] Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, it implements the steps of the foregoing frequency-domain-based unlearnable image privacy protection method.

[0081] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memory, CD-ROM, optical memory, etc.) that contain computer-usable program code.

[0082] The present invention is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the present invention. It should be understood that each flow and / or block in the flowchart and / or block diagram, as well as the combination of flows and / or blocks in the flowchart and / or block diagram, can be realized by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate means for realizing the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0083] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured article including instruction means, and the instruction means realizes the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0084] These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process. Therefore, the instructions executed on the computer or other programmable device provide steps for realizing the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0085] The above embodiments are only used to illustrate the technical idea of the present invention, and the protection scope of the present invention cannot be limited thereby. Any modification made on the basis of the technical solution according to the technical idea proposed by the present invention falls within the protection scope of the present invention.

Claims

1. A frequency-domain-based non-learnable image privacy protection method, characterized in that It includes the following steps: Step 1, for the original image to be protected, convert it from the RGB color space to the YUV color space to obtain a YUV color space image; Step 2, for the Y, U, and V channels of the YUV color space image, use the discrete cosine transform to transform each channel from the spatial domain to the frequency domain to obtain the DCT coefficient table of each channel; Step 3: Initialize a WGAN generator for each of the Y, U, and V channels and Initialize a WGAN discriminator D based on the original image to be protected 1 and a classifier C 1 ; The WGAN discriminator and the classifier are updated simultaneously, and the WGAN generator is updated once every m simultaneous updates of the WGAN discriminator and the classifier, where m ≥ 1; Step 4, for the i-th iteration of the WGAN discriminator and classifier, input the corresponding DCT coefficient table into the current WGAN generator for each channel to obtain the perturbed DCT coefficient table for each channel and Add each perturbed DCT coefficient table to the DCT coefficient table for the corresponding channel to obtain the new DCT coefficient table for each channel and Step 5, for the new DCT coefficient tables of each channel and perform inverse discrete cosine transform, convert each channel from the frequency domain back to the spatial domain, and perform channel merging to obtain a new YUV color space image. Convert the new YUV color space image back to the RGB color space to obtain the non-learnable image of the i-th iteration Step 6, input the unlearnable image of the i-th iteration into the WGAN discriminator D i-1 and the classifier C i-1 , and calculate the loss functions of the WGAN discriminator D i-1 and the classifier C i-1 respectively and Use and to update the WGAN discriminator D i-1 and the classifier C i-1 respectively, and obtain the WGAN discriminator D i and the classifier C i ; Step 7, for the j-th iteration of the WGAN generator, input the corresponding DCT coefficient table into the WGAN generator of each channel, and obtain the non-learnable image of the j-th iteration according to the same operations in Steps 4 - 5 and Input it into the current WGAN discriminator and the current classifier, calculate the loss functions of the current WGAN discriminator and the classifier respectively, calculate the loss function of the WGAN generator using the loss functions of the current WGAN discriminator and the classifier, and update the WGAN generator using the optimizer Input into the current WGAN discriminator and the current classifier, calculate the loss functions of the current WGAN discriminator and the classifier respectively, calculate the loss function of the WGAN generator using the loss functions of the current WGAN discriminator and the classifier, and update the WGAN generator using the optimizer and to obtain the WGAN generator and Step 8, determine whether the WGAN generator has reached the preset number of iterations. If so, use the WGAN generator of the last iteration and as the trained perturbation generator; otherwise, let i = i + 1 and return to Step 4; Step 9, use the trained perturbation generator to generate an unlearnable image for the image to be protected according to the same operations in Steps 4 - Step 5.

2. The frequency-domain-based unlearnable image privacy protection method according to claim 1, characterized in that The WGAN discriminator D i , the classifier C i , and the WGAN generator and are all multi-layer convolutional neural networks, and the classifier C i , the WGAN generator and all include residual modules.

3. The frequency-domain-based unlearnable image privacy protection method according to claim 1, characterized in that In the said Step 2, the formula of the discrete cosine transform is as follows: Among them, represents the value of the matrix obtained by discrete cosine transform at coordinates [u, v], N represents the image size, and x[k, m] represents the pixel value of the image at coordinates [k, m].

4. The method for protecting image privacy that cannot be learned based on the frequency domain according to claim 3, wherein In the said Step 5, the formula of the inverse discrete cosine transform is as follows: Among them, represents the pixel value of the matrix obtained by the inverse discrete cosine transform at the coordinates [k, m].

5. The frequency-domain-based non-learnable image privacy protection method according to claim 1, wherein In the said Step 6, the loss function of the WGAN discriminator is: Among them, l D represents the WGAN discriminator loss, represents the expectation of the sample x of the true data distribution , is the probability distribution of the true data, x is a sample randomly drawn from this distribution, is the discrimination result of the discriminator on the sample x, represents the expectation of the unlearnable image x generated by the generator ue , is the probability distribution of the unlearnable image, is the discrimination result of the discriminator on the unlearnable image x ue , λ represents the penalty coefficient, represents the expectation of the interpolated unlearnable sample , is the interpolation distribution between the true data and the data generated by the generator, is the interpolated unlearnable image after adding Gaussian noise, represents taking the gradient with respect to , represents the discrimination result of the discriminator on the unlearnable image ; The loss function of the classifier is: where l C represents the classifier loss, y represents the true label, represents the label predicted by the classifier; The loss function of the WGAN generator is: Among them, l G represents the WGAN generator, and α and β are the weights of the two parts of the loss respectively.

6. The frequency-domain-based non-learnable image privacy protection method according to claim 1, characterized in that In the said Step 7, the optimizer is Adam.

7. A computer device, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the frequency-domain-based unlearnable image privacy protection method according to any one of claims 1 to 6.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the frequency-domain-based unlearnable image privacy protection method according to any one of claims 1 to 6.

Images