Method for generating protection image and program

Abstract

To provide a method for generating a protection image and a program capable of degrading the quality of an image output by a learned model.SOLUTION: The method for generating a protection image includes: a latent representation acquisition step of inputting a target image as an input image to an encoder of a variational auto-encoder to acquire a latent representation; a reconstructed image acquisition step of inputting the latent representation to a decoder of the variational auto-encoder to acquire a reconstructed image; a difference calculation step of calculating the difference between the reconstructed image and the input image; a perturbation calculation step of calculating a perturbation to be added to the input image so that the difference with respect to the target image becomes large; and an image acquisition step of updating the target image on the basis of the perturbation to acquire an updated image.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present invention relates to a method and a program for generating a protected image. [Background technology]

[0002] In recent years, artificial intelligence (AI) technologies, including deep neural networks (DNNs), have made remarkable progress. In particular, generative AI, which generates images and text based on instructions in natural language, is expected to have a wide range of applications, and many image generation services have been released.

[0003] However, these image generation AIs can sometimes cause problems with inappropriate use. For example, images published online are sometimes used as conditions for image generation without the permission of the rights holder, resulting in the unauthorized use of images from the internet. Furthermore, in addition to infringement of rights, images containing false information can be generated, which could be used by malicious users on social media to spread fake news.

[0004] Therefore, there is a need for countermeasures that can prevent unauthorized content use by image generation AI. For example, the method for generating one-pixel attack samples disclosed in Patent Document 1 includes the steps of traversing each pixel value at pixel point position P in the recognition target image, obtaining pixel value V' when the recognition probability of each value in a preset neural network N is lowest, and correcting the pixel value at pixel position P in the recognition target image X to V' to obtain an adversarial sample X'. This configuration modifies one pixel point in the recognition target image to reduce the probability that the neural network will accurately recognize the image. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Publication No. 2020-170495 Summary of the Invention [Problem to be solved by the invention]

[0006] The method of generating attack samples as disclosed in Patent Document 1 does not provide sufficient content protection, and there is a demand for greater content protection.

[0007] The present invention has been made in consideration of the above-mentioned conventional circumstances, and one object of the present invention is to provide a method and program for generating a protected image that can reduce the quality of an image output by a trained model. [Means for solving the problem]

[0008] The method for generating a protected image of the present disclosure includes: a latent representation acquisition step of inputting a target image as an input image into an encoder of a variational autoencoder to acquire a latent representation; a reconstructed image acquisition step of inputting the latent representation into a decoder of the variational autoencoder to acquire a reconstructed image; a difference calculation step of calculating a difference between the reconstructed image and the input image; a perturbation calculation step of calculating a perturbation to be added to the input image so as to increase the difference with respect to the target image; an image acquisition step of updating the target image based on the perturbation to acquire an updated image; Equipped with The updated image acquired in the image acquisition process is used as the input image for the latent expression acquisition process, and the updated image acquired by repeatedly performing the latent expression acquisition process, the reconstructed image acquisition process, the difference calculation process, the perturbation calculation process, and the image acquisition process is generated as a protected image of the target image.

[0009] The program of the present disclosure causes a computer to execute the above-described method for generating a protected image. [Effects of the Invention]

[0010] The techniques disclosed herein can degrade the quality of the images output by the trained model. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a block diagram showing the electrical configuration of an image processing apparatus according to the first embodiment. [Figure 2] Figure 2 shows a schematic diagram of the structure of the latent diffusion model. [Figure 3] FIG. 3 is a diagram illustrating an image generation method using a latent diffusion model. [Figure 4] FIG. 4 is a diagram illustrating relearning by LoRA. [Figure 5] Figure 5 is a diagram illustrating image generation using the LoRA trained model. [Figure 6] Figure 6 shows an overview of our image protection method using untargeted adversarial perturbations to a variational autoencoder. [Figure 7] FIG. 7 is a flowchart showing the flow of the protected image generation control executed by the image processing device. [Figure 8] Figure 8 shows an algorithm (pseudocode) for generating non-targeted protection samples for a variational autoencoder. [Figure 9] Figure 9 shows the algorithm (pseudocode) for Lp norm projection (project() function). [Figure 10] FIG. 10 is a diagram illustrating the latent expression acquisition step in the protected image generation step. [Figure 11] FIG. 11 is a diagram illustrating the reconstructed image acquisition process. [Figure 12] FIG. 12 is a diagram illustrating the difference calculation process. [Figure 13] FIG. 13 is a diagram illustrating the perturbation calculation process. [Figure 14] FIG. 14 is a diagram illustrating the image acquisition process. [Figure 15]Fig. 15(A) is a diagram showing an example of a target image to be protected, and Fig. 15(B) is a diagram showing an example of a protection image for the target image of Fig. 15(A). [Figure 16] Figure 16(A) is an image output using a model trained with LoRA on the target image in Figure 15(A). Figure 16(B) is an image output using a model trained with LoRA on the protected image in Figure 15(B). DETAILED DESCRIPTION OF THE INVENTION

[0012] In the following, embodiments of the present disclosure are listed and illustrated. [1] A latent representation acquisition step in which a target image is input as an input image into an encoder of a variational autoencoder to acquire a latent representation; a reconstructed image acquisition step of inputting the latent representation into a decoder of the variational autoencoder to acquire a reconstructed image; a difference calculation step of calculating a difference between the reconstructed image and the input image; a perturbation calculation step of calculating a perturbation to be added to the input image so as to increase the difference with respect to the target image; an image acquisition step of updating the target image based on the perturbation to acquire an updated image; Equipped with the updated image acquired in the image acquisition step is used as the input image of the latent expression acquisition step, and the updated image acquired by repeatedly performing the latent expression acquisition step, the reconstructed image acquisition step, the difference calculation step, the perturbation calculation step, and the image acquisition step is generated as a protected image of the target image; How to generate a protected image.

[0013] According to the configuration in [1] above, a variational autoencoder is used to obtain a reconstructed image from a target image to be protected. Then, using this reconstructed image, a perturbation to be added to the input image can be calculated so that the difference with the target image becomes large. By using the updated image obtained by adding this perturbation to the target image as the protected image, the quality of the image output by the model trained using the protected image can be reduced.

[0014] [2] In the image acquisition step, an image obtained by adding the perturbation to the input image is compared with the target image, the perturbation is adjusted so that it does not exceed a predetermined set value, and the adjusted perturbation is added to the target image to acquire the updated image. The method for generating a protection image according to [1].

[0015] According to the configuration of [2] above, the magnitude of change in the updated image when the process of acquiring the updated image is performed multiple times can be kept below a certain value (set value). Therefore, in the process of acquiring the updated image, even small changes can be prevented from becoming significant by sending the process back multiple times.

[0016] [3] A program that causes a computer to execute the method for generating a protection image according to [1] or [2].

[0017] According to the configuration [3] above, the same effect as the effect of [1] above can be achieved.

[0018] First Embodiment (Image processing device) An image processing device 10 of the present disclosure will be described with reference to Fig. 1. The image processing device 10 is a general information processing device, and performs a learning process using a latent diffusion model (described later), an image generation process using a latent diffusion model (described later), and a protected image generation process. The image processing device 10 is an information processing device (computer) that executes a program for generating a protected image. The image processing device 10 stores various types of image data and displays images based on the stored various types of image data.

[0019] As shown in FIG. 1, the image processing device 10 includes a control unit 11, a storage unit 12, an input unit 13, and a display unit 14.

[0020] The control unit 11 controls the entire image processing device 10. The control unit 11 includes an arithmetic unit such as an MCU (Micro Processor Unit) and other peripheral circuits, and can perform various controls and calculations. The control unit 11 performs control based on programs stored in the storage unit 12, for example.

[0021] The storage unit 12 is configured with a known storage device such as a semiconductor memory, and corresponds to, for example, a RAM, a ROM, a non-volatile memory, etc. The storage unit 12 stores various programs executed by the control unit 11, etc.

[0022] The input unit 13 is an input interface for inputting data, and is configured by, for example, a keyboard, a touch panel, a mouse, a scanner, etc. The input unit 13 receives input of a target image, etc., which will be described later.

[0023] The display unit 14 is configured as a known image display device such as a liquid crystal display, an organic electroluminescence display, a touch panel, etc. The display unit 14 performs display based on various image data stored in the storage unit 12.

[0024] (latent diffusion model) The Latent Diffusion Model (LDM) is one of the most representative image generation models. Figure 2 shows an overview of the structure of the Latent Diffusion Model. As shown in Figure 2, the Latent Diffusion Model has a structure in which a Diffusion Model (DM) is sandwiched between the encoder and decoder of a Variational Autoencoder (VAE).

[0025] In the training of the diffusion model, first, the data is gradually diffused by adding random noise to the original data. By repeating this process enough times, the data becomes completely random noise. This process is called the diffusion process.

[0026] Next, in the de-diffusion process, the random noise is gradually removed to restore the original data. In this de-diffusion process, a neural network is trained to correctly estimate what kind of noise has been added to the data. For example, a structure based on U-Net incorporating a Transformer is used for the neural network.

[0027] In image generation using a well-trained diffusion model, perfect noise is used as the initial samples and fed into the neural network for the de-diffusion process. The resulting noise is then subtracted from the input samples, completing one stage of de-noising. This procedure is repeated for a number of time steps, ultimately producing a meaningful image from the noise.

[0028] Diffusion models can generate higher quality data than conventional methods. However, the learning and inference of diffusion models are computationally expensive, requiring large amounts of computational resources and training data. Therefore, the latent diffusion model calculates a latent representation that reduces the dimensionality of the input image using a variational autoencoder and trains the diffusion model, enabling learning and inference with limited computational resources while maintaining the quality and accuracy of the generated images.

[0029] Figure 3 is a schematic diagram explaining the image generation method using the latent diffusion model. As shown in Figure 3, images are created by using, for example, noise generated from random numbers as a basis, tokenizing prompts and embedding them in fixed-length vectors, and using this as conditioning.

[0030] For example, in the field of image generation, there is a method that uses LoRA (Low-Rank Adaptation) additional learning for stable diffusion (an image generation model using a latent diffusion model). LoRA is a method that gives a model the ability to perform a specific task by retraining a trained generative model using a small amount of additional data. For example, when applying LoRA to an image generation model, by using images of works by a specific artist, the image generation model can be made to reproduce the artist's style.

[0031] For example, we retrain LoRA to associate the image style with the prompt "irsty" using the image shown in Figure 4. When we give the Stable Diffusion model obtained in this way the prompt "irsty rabbit," we get the image shown in Figure 5.

[0032] (Protected image generation method) A method for generating a protected image according to the present disclosure will now be described. The method for generating a protected image according to the present disclosure protects a target image (original image) to be protected using non-targeted adversarial samples for a variational autoencoder. FIG. 6 is a diagram showing an overview of the method for generating a protected image according to the present disclosure. As shown in FIG. 6, a perturbation is calculated to increase the difference between the reconstructed image output by the variational autoencoder and the target image. FIG. 7 is a flowchart showing the flow of control for generating a protected image executed by the image processing device 10.

[0033] The program of the present disclosure is a program for causing a computer (the control unit 11 of the image processing device 10) to execute the method for generating a protected image of the present disclosure. Figures 8 and 9 show pseudocode for causing a computer (the control unit 11 of the image processing device 10) to execute the method for generating a protected image of the present disclosure. Those skilled in the art will understand that the pseudocode represents various processes for causing a computer to execute the program of the present disclosure.

[0034] The protected image generation method of the present disclosure includes a latent expression acquisition step, a reconstructed image acquisition step, a difference calculation step, a perturbation calculation step, and an image acquisition step. The protected image generation method of the present disclosure is performed in the order of the latent expression acquisition step, the reconstructed image acquisition step, the difference calculation step, the perturbation calculation step, and the image acquisition step. Each step will be described below.

[0035] In the latent representation acquisition process, a target image is input as an input image into the encoder of a variational autoencoder to acquire a latent representation (step S2 in Figure 7). Note that before step S2 in Figure 7, i, the number of image updates (described later), is initialized to 1 (step S1). The latent representation acquisition process corresponds to steps 1-4 in the pseudocode in Figure 8. As shown in Figure 10, a target image is input as an input image to a variational autoencoder, which may constitute part of an image generation model, to acquire a latent representation. The image is treated as a vector in the format of (number of channels, height, width). The input image is, for example, a vector represented as (512,512,3). The target image is subjected to preprocessing such as resizing and normalization according to the specifications of the variational autoencoder. The latent representation is, for example, a vector represented as (64,64,4).

[0036] In the reconstructed image acquisition process, the latent representation is input to the decoder of the variational autoencoder to obtain a reconstructed image (step S3 in Figure 7). The reconstructed image acquisition process corresponds to step 5 in the pseudocode in Figure 8. As shown in Figure 11, the latent representation obtained in the latent representation acquisition process is input to the decoder of the variational autoencoder to obtain a reconstructed image. The reconstructed image is, for example, a vector represented by (512, 512, 3). The variational autoencoder has a structure aimed at reducing the dimensionality of input information and performs a lossy image compression / decompression algorithm. Therefore, an image that is perceptually very similar to the target image is obtained as the output image.

[0037] In the difference calculation step, the difference (error) between the reconstructed image and the input image is calculated (step S4 in FIG. 7). The difference calculation step corresponds to step 6 in the pseudocode in FIG. 8. As shown in FIG. 12, the error between the reconstructed image obtained in the reconstructed image acquisition step and a pre-prepared input image is calculated. The reconstructed image and the input image are represented by vectors (number of channels, height, width). The error is found using a pre-defined error function (loss function). The error function is a function that takes a smaller value the more similar the two vectors are, and one example is the mean square error.

[0038] The perturbation calculation step calculates a perturbation (adversarial perturbation) to be added to the input image so as to increase the difference with respect to the target image (step S5 in Figure 7). The perturbation calculation step corresponds to part of step 7 in the pseudocode in Figure 8. As shown in Figure 13, the error calculated in the difference calculation step is subjected to automatic differentiation using the error backpropagation method on the variational autoencoder model to obtain the gradient of the error with respect to the input image. Furthermore, the direction (positive or negative sign) of the obtained gradient is determined and used as the perturbation. The gradient is a collection of partial derivatives with respect to the loss (difference) for the input image. In other words, the gradient is a list of partial derivative values ​​for each element of the input image. The partial derivative of each element indicates "how much influence that element has on the loss." If the value of a partial derivative is positive, reducing that element reduces the loss. Conversely, if the value of the partial derivative is negative, increasing that element reduces the loss. The perturbation is a vector with the same dimensions (number of channels, height, and width) as the input image, but each element takes the value -1, 0, or 1. This is a similar procedure to the FGSM (Fast Gradient Sign Method), which is a method for generating adversarial samples for classification models.

[0039] In the image acquisition process, the target image is updated based on the perturbation to obtain an updated image (step S6 in FIG. 7). The image acquisition process corresponds to steps 7-10 in the pseudocode in FIG. 8. As shown in FIG. 14, in the image acquisition process, an image obtained by adding the perturbation to the input image is compared with the target image, the perturbation is adjusted so that it does not exceed a predetermined set value, and the adjusted perturbation is added to the target image to obtain an updated image.

[0040] Specifically, loss i Input image S for i As the gradient of ∇S i loss i Calculate ∇S i loss i By inputting this into the sign() function, we can extract only the sign. That is, ∇S i loss i All values ​​of are converted to -1 or 1. The input to the sign() function is multiplied by a coefficient α and the result is the input image S i Add this to S i adv (Step 7 in the pseudocode in FIG. 8). The absolute value of the magnitude changed by one processing is determined by α.

[0041] S i adv The amount of change in each time is controlled by α, but even if the change is small, it can become significant if the process is repeated many times. Therefore, a process is performed to suppress the magnitude of the change when multiple processes are performed below a certain value. First, the S obtained in step 7 i adv By subtracting the target image (original image S0) from the original image S1, the difference is calculated for each pixel element, and the result is called diff i (Step 8). Then, the project() function is used to calculate the difference diff. i The Lp norm of is projected so that it does not exceed ε, and this is called η (Step 9).

[0042] Here, in the Lp norm, p is a real number equal to or greater than 1, and the Lp norm for a vector x is formulated by the following equation (1). Equation (1) is the equation when equation (2) is given below.

[0043]

number

[0044]

number

[0045] By adding the obtained η to the target image S0, a perturbation is added and an updated image is obtained in which the perturbation is suppressed to ε or less (step 10). i+1 This becomes the input image for the next process.

[0046] The updated image acquired in the image acquisition step is used as an input image for the latent expression acquisition step, and the latent expression acquisition step, reconstructed image acquisition step, difference calculation step, perturbation calculation step, and image acquisition step are repeated to acquire an updated image, which is generated as a protected image of the target image. Steps S2-S8 in FIG. 8 are repeated until i becomes N. That is, steps 3-10 in the pseudo code in FIG. 8 are repeated. Then, the updated image S output by the Nth processing is Nis obtained as a protected image (step S9 in FIG. 8).

[0047] (Evaluation experiment) An evaluation experiment was conducted on a protected image protected using the protected image generation method of the present disclosure. Figure 15(A) shows a target image. Figure 15(B) shows a protected image of the target image shown in Figure 15(A) protected using the above generation method. The number of times the image was updated (the number of times steps 3-10 in the pseudocode in Figure 8 were repeated) was 5,000. It is difficult to visually determine the difference between the target image (Figure 15(A)) and the protected image (Figure 15(B)).

[0048] Figure 16(A) shows an image obtained by training the model using LoRA with the target image as "kmgic cat" and the prompt "kmgic cat wearing blue sweater." An image similar to the image used for training was obtained.

[0049] Figure 16(B) shows an image obtained by training the model using LoRA with the protected image as a "kmgic cat" and then using the prompt "kmgic cat wearing a blue sweater." The output from the model trained on the protected image contains unnatural patterns (white areas) with relatively high saturation.

[0050] (Operation and effect of the first embodiment) The protected image generation method of the first embodiment includes a latent representation acquisition step of inputting a target image as an input image into the encoder of a variational autoencoder to acquire a latent representation, a reconstructed image acquisition step of inputting the latent representation into the decoder of the variational autoencoder to acquire a reconstructed image, a difference calculation step of calculating the difference between the reconstructed image and the input image, a perturbation calculation step of calculating a perturbation to be added to the input image so as to increase the difference with respect to the target image, and an image acquisition step of updating the target image based on the perturbation to acquire an updated image. This generation method generates an updated image as a protected image of the target image by repeatedly performing the latent representation acquisition step, reconstructed image acquisition step, difference calculation step, perturbation calculation step, and image acquisition step, using the updated image acquired in the image acquisition step as an input image for the latent representation acquisition step.

[0051] With this configuration, a variational autoencoder is used to obtain a reconstructed image from the target image to be protected. This reconstructed image can then be used to calculate a perturbation to be added to the input image so that the difference with the target image increases. By adding this perturbation to the target image, the updated image can be used as the protected image, thereby reducing the quality of the image output by the model trained using the protected image.

[0052] By using the above-mentioned method for generating protected images, it is possible to reduce the quality of the image output when a protected target image, which is difficult for the human eye to perceive, is input into an image generation AI. This prevents the unauthorized use of protected target images. It also makes it possible to confirm that the original target image was used for training purposes against the rights holder's intentions. As described above, it is possible to prevent image content from being used for model training without the rights holder's permission.

[0053] Furthermore, in the protection image generation method of the first embodiment, in the image acquisition process, an image obtained by adding a perturbation to an input image is compared with the target image, the perturbation is adjusted so that it does not exceed a predetermined set value, and the adjusted perturbation is added to the target image to acquire an updated image.

[0054] This configuration makes it possible to suppress the magnitude of change in the updated image when the process of acquiring the updated image is performed multiple times to a certain value (set value) or less. Therefore, in the process of acquiring the updated image, even small changes can be prevented from becoming significant by sending the process back multiple times.

[0055] <Other embodiments> The present invention is not limited to the embodiments described above and illustrated in the drawings, and the following embodiments are also included within the technical scope of the present invention. Furthermore, the various features of the above-mentioned embodiments and the embodiments to be described later may be combined in any manner as long as they are not contradictory.

[0056] In the image acquisition process in the first embodiment, in the calculation of the project() function in step 9 of the pseudo code in FIG. 8 (pseudo code in FIG. 9), p of the Lp norm is set to a real number of 1 or more, but the difference diff i The maximum absolute value of each of the above may be set so as not to exceed ε.

[0057] The number of times N that steps 3-10 of the pseudo code in FIG. 8 are repeated in the image generating process in the first embodiment is not particularly limited, and is preferably 5000 times, for example.

[0058] It should be noted that the embodiments disclosed herein should be considered to be illustrative in all respects and not restrictive. The scope of the present invention is not limited to the embodiments disclosed herein, but is intended to include all modifications within the scope indicated by the claims or within the scope equivalent to the claims. [Explanation of symbols]

[0059] 10: Image Processing Device 11: Control section 12: Storage section 13: Input section 14: Display section

Claims

1. a latent representation acquisition step of inputting a target image as an input image into an encoder of a variational autoencoder to acquire a latent representation; a reconstructed image acquisition step of inputting the latent representation into a decoder of the variational autoencoder to acquire a reconstructed image; a difference calculation step of calculating a difference between the reconstructed image and the input image; a perturbation calculation step of calculating a perturbation to be added to the input image so as to increase the difference with respect to the target image; an image acquisition step of updating the target image based on the perturbation to acquire an updated image; Equipped with The updated image acquired in the image acquisition step is used as the input image of the latent expression acquisition step, and the updated image acquired by repeatedly performing the latent expression acquisition step, the reconstructed image acquisition step, the difference calculation step, the perturbation calculation step, and the image acquisition step is generated as a protected image of the target image. How to generate a protected image.

2. In the image acquisition step, an image obtained by adding the perturbation to the input image is compared with the target image, the perturbation is adjusted so that it does not exceed a predetermined set value, and the adjusted perturbation is added to the target image to acquire the updated image. The method for generating a security image according to claim 1 .

3. 3. A program that causes a computer to execute the protection image generating method according to claim 1.

Images