Information processing device, server, information processing method, and program

Abstract

To improve convenience related to use of a diffusion model.SOLUTION: An information processing device capable of generating an adversarial image in a diffusion model includes a control unit configured to generate the adversarial image based on a first image and a watermark image including identification information. The adversarial image is an image in which identification information can be visually recognized in a second image when the second image is generated based on the diffusion model and the adversarial image.SELECTED DRAWING: Figure 1-1

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device, a server, an information processing method, a program, and the like. [Background technology]

[0002] With the rapid development and widespread use of generative AI based on diffusion models (DMs), copyright infringement of products generated by generative AI has become a concern. For example, Non-Patent Document 1 discloses a diffusion model that can embed an invisible digital watermark in an image. It also discloses a model that recovers a signature linked to an image by detecting the digital watermark from the embedded image. However, the method disclosed in Non-Patent Document 1 requires re-learning of the diffusion model, which requires time and computational resources for processing. Also, a mechanical detection process is required to detect the invisible digital watermark in order to restore the signature.

[0003] Also, for example, Non-Patent Document 2 discloses an algorithm for generating an adversarial image to protect an original image from learning or imitation using a diffusion model. However, although the method disclosed in Non-Patent Document 2 can prevent the generation of imitative images using a diffusion model, it cannot embed information about the author of the image. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Pierre Fernandez, et al. “The Stable Signature: Rooting Watermarks in Latent Diffusion Models”, 2023. [Non-patent document 2] Chumeng Liang, et al. “Adversarial Example Does Good: Preventing Painting Imitation from Diffusion Models via Adversarial Examples”, 2023. Summary of the Invention [Problem to be solved by the invention]

[0005] The present invention has been made in light of the above-mentioned technical background, and aims to provide an information processing method for generating an adversarial image in which a visible digital watermark is embedded in an image generated by a diffusion model, without retraining the diffusion model. [Means for solving the problem]

[0006] According to a first aspect of the present invention, an information processing device capable of generating an adversarial image in a diffusion model includes a control unit that generates the adversarial image based on a first image and a watermark image including identification information, and the adversarial image is an image in which the identification information becomes visible in a second image when the second image is generated based on the diffusion model and the adversarial image. According to a second aspect of the present invention, an information processing method in an information processing device capable of generating an adversarial image in a diffusion model generates an adversarial image based on a first image and a watermark image including identification information, and the adversarial image is an image in which the identification information becomes visible in a second image when the second image is generated based on the diffusion model and the adversarial image. According to a third aspect of the present invention, a program executed by an information processing device capable of generating an adversarial image in a diffusion model generates the adversarial image by a control unit of the information processing device based on a first image and a watermark image including identification information, and the adversarial image is an image in which the identification information becomes visible in a second image when the second image is generated based on the diffusion model and the adversarial image. [Brief explanation of the drawings]

[0007] [Figure 1-1] FIG. 1 is a diagram showing an example of the configuration of an information processing device according to a first embodiment. [Figure 1-2] 4 is a flowchart showing an example of the flow of processing executed by the information processing device according to the first embodiment. [Figure 1-3] FIG. 1 is a diagram showing an example of an image generated from an adversarial image using a diffusion model by the information processing device according to the first embodiment. [Figure 1-4] FIG. 1 is a diagram showing an example of an image generated by a diffusion model that has learned an adversarial image by the information processing device according to the first embodiment. [Figure 2-1] FIG. 10 is a diagram showing an example of the system configuration of a communication system according to a second embodiment. [Figure 2-2] FIG. 10 is a diagram showing an example of functions realized by a control unit of a server according to a second embodiment. [Figure 2-3] FIG. 10 is a diagram showing an example of information stored in a storage unit of a server according to a second embodiment. [Figure 2-4] FIG. 10 is a diagram showing an example of account registration data according to the second embodiment. [Figure 2-5] FIG. 10 is a diagram showing an example of functions realized by a control unit of a terminal according to a second embodiment. [Figure 2-6] FIG. 10 is a diagram showing an example of information stored in a storage unit of a terminal according to a second embodiment. [Figure 2-7] FIG. 10 is a diagram showing an example of a screen displayed on a display unit of a terminal according to a second embodiment. [Figure 2-8] 10 is a flowchart showing an example of the flow of processing executed by each device according to the second embodiment. [Figure 3-1] FIG. 11 is a diagram showing an example of a screen displayed on a display unit of a terminal according to a third embodiment. [Figure 3-2] FIG. 11 is a diagram showing an example of a screen displayed on a display unit of a terminal according to a third embodiment. [Figure 3-3] 10 is a flowchart showing an example of the flow of processing executed by each device according to the third embodiment. [Figure 3-4] 13 is a table showing an example of destinations of hostile image use warning information according to the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0008] <Compliance with legal matters> It should be noted that the disclosures set forth herein are subject to compliance with the laws of the country of implementation, such as communications privacy, as required for the implementation of the disclosures.

[0009] <Embodiment> In this specification, for ease of understanding, there are places where the term "for example" is used, but please note that not only those places but also the entire embodiment described below are not limited to the content of the description.

[0010] An embodiment for implementing a program etc. according to the present disclosure will be described with reference to the drawings.

[0011] The production of a terminal of the claimed invention (terminal of the claimed invention) may include, for example, the concept that a state is created in which the functions of the claimed invention can be realized (a state in which the claimed invention can be executed) on a terminal owned (possessed) by a user by receiving (or receiving and storing) a program (for example, an application program) described in this specification.

[0012] Furthermore, the production of the system of the invention claimed in the present application (the system of the invention of the present application) may include, for example, the concept that a state in which the functions of the invention of the system claimed in the present application can be realized (a state in which the invention claimed in the present application can be executed) is created by receiving a program described in this specification (for example, an application program) transmitted from a server included in the system of the present application at a terminal included in the system of the present application (or by storing the received program in the terminal).

[0013] In this specification, a system may be, for example, configured to include a plurality of devices. The plurality of devices may be a combination of devices of the same type, a combination of devices of different types, or a combination of devices of the same type and devices of different types. A system can also be thought of as, for example, a plurality of devices working together to perform some kind of processing.

[0014] Furthermore, a system relating to a client (client device) and a server can be considered to be, for example, at least one of the following: (1) Terminals and servers (2) Server (3) Terminal

[0015] (1) is, for example, a system including at least one terminal and at least one server. One example of this is a client-server system.

[0016] The server is configured by the following devices, for example, and may be a single device or a combination of multiple devices.

[0017] Specifically, a server is configured to have, for example, at least one processor (for example, CPU: Central Processing Unit, GPU: Graphics Processing Unit, APU: Accelerated Processing Unit, DSP: Digital Signal Processor (for example, ASIC: Application Specific Integrated Circuit, FPGA: Field Programmable Gate Array), etc.), computer device (processor + memory), control device, arithmetic device, processing device, etc., and may be configured to have multiple of the same type of device (for example, CPU + CPU, homogeneous multi-core processor, etc.), or multiple of different types of device (for example, CPU + DSP, heterogeneous multi-core processor, etc.), or may be a combination of multiple devices (for example, processor + computer device, processor + arithmetic device, multiple devices made heterogeneous, etc.). The processor may be a virtual processor.

[0018] Furthermore, when a server performs some processing, if the server is configured with a single device, the processing described in the embodiments is performed by the single device. Furthermore, if the server is configured with multiple devices, one device may perform some of the processing, and the other device may perform other processing. For example, if the server is configured with a processor and an arithmetic device, the processor may perform a first processing, and the arithmetic device may perform a second processing. Furthermore, when a plurality of devices are used, the devices may be located at positions physically separated from one another.

[0019] Furthermore, the server functions may be provided in the form of PaaS, IaaS, or SaaS in cloud computing, for example. Some or all of the processes described in this specification may be implemented as a program included in an application installed on a terminal from a server. Furthermore, the manufacturer and manager of the terminal, the manufacturer and manager of the application, and the manufacturer and manager of the server may be different entities (businesses), or some or all may be the same entity (business).

[0020] The control unit of the system can be at least one of the control unit of the terminal and the control unit of the server. That is, for example, the control unit of the system can be any of (1A) only the control unit of the terminal, (1B) only the control unit of the server, or (1C) both the control unit of the terminal and the control unit of the server.

[0021] Furthermore, the control and processing (hereinafter collectively referred to as "control, etc.") performed by the control unit of the system may be performed by (1A) only the control unit of the terminal, (1B) only the control unit of the server, or (1C) both the control unit of the terminal and the control unit of the server. In addition, in (1C), for example, some of the controls performed by the control unit of the system may be performed by the control unit of the terminal, and the remaining controls may be performed by the control unit of the server. In this case, the allocation (allocation) of the controls may be equal or may be allocated in different proportions.

[0022] Furthermore, when referring to the communication unit of a server, if the server is configured with a single device, it may refer to the communication unit itself that the single device has, or if the server is configured with multiple devices, it may be configured to include each communication unit that each device has. For example, if a server comprises a first device and a second device, and the first device has a first communication unit and the second device has a second communication unit, the communication unit of the server may be considered to include the first communication unit and the second communication unit.

[0023] (2) can be, for example, a system consisting of multiple servers (hereinafter referred to as a "server system"). In this case, the configuration of each server can be similarly applied to the configuration described above.

[0024] The control etc. performed by the server system may be performed by only one of the multiple servers (2A), by only the other servers (2B), or by both the one server and the other servers (2C). In addition, in (2C), for example, one server may perform some of the control, etc., performed by the server system, and another server may perform the remaining control, etc. In this case, the allocation (allocation) of the control, etc. may be equal or may be allocated in different proportions.

[0025] (3) can be, for example, a system consisting of multiple terminals. This system can be, for example, the following system. A system that gives server functions to terminals (distributed system). This can be realized, for example, using blockchain technology. A system in which terminals communicate wirelessly with each other. This can be achieved, for example, by using short-range wireless communication technology such as Bluetooth (registered trademark) to communicate in a P2P (peer-to-peer) format.

[0026] The above is not limited to the control unit, but also applies to each functional unit such as an input / output unit, a communication unit, a storage unit, and a clock unit that may be components of the system.

[0027] In the following embodiment, a system including a terminal and a server (a client-server system, for example) will be described as an example. It is also possible to apply the server system described in (2) above as the server.

[0028] Furthermore, instead of a system including a terminal and a server, it is also possible to apply a system that does not include a server, such as the system (3) above. In this case, the embodiment can be configured based on the above-mentioned blockchain technology, etc. Specifically, for example, data stored and managed in a server described in the following embodiment is stored on the blockchain. Then, a terminal generates a transaction to the blockchain, and when the transaction is approved on the blockchain, the data stored on the blockchain is updated.

[0029] It should be noted that even when the term "terminal" is used, this is not limited to the meaning of a terminal as a client device in a client server. That is, a terminal may include the concept of a device that is not in a client-server context.

[0030] Furthermore, in this specification, the expression "through a communication I / F" is used as appropriate. This may, for example, indicate that a device transmits and receives various information and data via a communication I / F (via a communication unit) based on the control of a control unit (such as a processor).

[0031] Furthermore, in this specification, when the terms "related to" or "related to" are used, for example, "B related to A" or "B related to A" may mean "B" that has some kind of relationship with "A." Specific examples of this will be described later.

[0032] Furthermore, in this specification, when a device performs processing on two or more objects, such as "sending A and B" or "receiving A and B," this may include performing "A" and "B" at the same time (hereinafter referred to as "simultaneous"), and performing "A" and "B" at different times (hereinafter referred to as "non-simultaneous"). For example, when referring to transmitting first information and second information, this may include both the concepts of transmitting the first information and the second information at the same time, and transmitting the first information and the second information at different times. In addition, taking into account the lag (time lag), "simultaneous" may include "almost simultaneously."

[0033] Note that even though "A" and "B" are performed at different times, this only needs to be done with "A" and "B" as the processing targets, and the purposes do not necessarily have to be the same. For example, when the first information and the second information are transmitted as described above, it is sufficient to transmit the first information and the second information, and this may include cases where the first information and the second information are transmitted for the same purpose, as well as cases where the first information and the second information are transmitted for different purposes.

[0034] Hereinafter, an example of an embodiment of the present invention will be described with reference to the drawings. In the description of the drawings, the same elements are denoted by the same reference numerals, and duplicated descriptions may be omitted. Furthermore, the components described in this embodiment are merely examples and are not intended to limit the scope of the present invention.

[0035] <First Example> The first embodiment is an embodiment in which an information processing device (which may also be called an image processing device, an image generation device, an adversarial image generation device, or an adversarial image generation unit) generates an adversarial image based on, for example, an original image and a watermark image. The contents described in the first embodiment can be similarly applied to any of the other embodiments and other modified examples.

[0036] FIG. 1-1 is a block diagram showing an example of a functional configuration of an information processing device 1 according to an aspect of the present embodiment. The information processing device 1 includes, for example, a perturbation control unit 2, a diffusion model unit 3, an adversarial image generation unit 4, a differential loss calculation unit 5, and an adversarial loss calculation unit 6. These are, for example, functional units (functional blocks) included in a control unit (control device) (not shown) of the information processing device 1. The control unit may also be called a processing unit (processing device).

[0037] The perturbation control unit 2 has the function of calculating a perturbation, which is a change that has a significant impact on the image (referred to as the "diffusion generated image") generated in the diffusion model unit 3 based on, for example, an original image 7 "x" and a watermark image 8 "m" input to the information processing device 1, while minimizing the impact on the original image "x." The perturbation control unit 2 is configured with an encoder-decoder model such as an autoencoder or U-Net. Hereinafter, the weight of the model of the perturbation control unit 2 (hereinafter, the "weight of the model" will be referred to as the "model weight") will be represented as "G", and the perturbation of the original image 7 "x" under the condition that the watermark image 8 "m" is given will be represented as "G(x|m)". The model weight can also be said to be the connection weight between neurons in each model.

[0038] The diffusion model unit 3 is configured with latent diffusion models such as DDPM (Denoising Diffusion Probabilistic Models) and Stable Diffusion. In the diffusion model, for example, a reference image "y" is converted into a latent representation in a diffusion process. Then, the latent representation is converted into a diffusion-generated image in a de-diffusion process. The diffusion model unit 3 may convert into a latent representation upon receiving a prompt. In the following, the model weight of the diffusion model unit 3 is represented as "θ", and the diffusion generated image generated when a reference image "y" is given is represented as "θ(y)".

[0039] The adversarial image generator 4 has a function of generating an adversarial image 9 "x'" based on, for example, a perturbation "G(x|m)" and an original image 7 "x". Here, the adversarial image 9 is an image that, when provided as an input to the diffusion model unit 3 as a reference image or an image for additional learning, has an unusual effect on the generation of a diffusion-generated image in the diffusion model unit 3 (interfering with the generation or adding noise, etc., unintended by the generator).

[0040] The differential loss calculation unit 5 has a function of calculating the differential loss “Ldiff” to be used for adjusting the model weight of the perturbation control unit 2, for example, based on the original image 7 “x”, the adversarial image 9 “x’”, and the watermark image 8 “m”. The adversarial loss calculation unit 6 also has a function of calculating, for example, the diffusion generated image “θ(x′)” and the adversarial loss “Ladv” to be used for adjusting the model weight of the perturbation control unit 2.

[0041] The original image 7 "x" is a tensor configured of the original image 7, which is, for example, a three-channel RGB image (color image) of "M×M" pixels ("M" is a natural number). The original image 7 "x" may be a tensor composed of a plurality of "M x M" pixel original images 7. The original image 7 may also be a grayscale or black and white binary image (one channel image).

[0042] The watermark image 8 "m" is a tensor made up of the watermark image 8, which is, for example, a three-channel image of "MxM" pixels. The watermark image 8 “m” may be a tensor configured by a one-channel image of “M×M” pixels, for example. The watermark image 8 and each original image 7 may be different sizes.

[0043] The adversarial image 9 "x'" is, for example, a tensor composed of the adversarial image 9 composed of a three-channel RGB image of "M x M" pixels. Note that the adversarial image 9 "x'" may be a tensor composed of any number of adversarial images 9. The adversarial image 9 may also be a grayscale or black and white binary image. The original image 7 and the adversarial image 9 may also be different sizes.

[0044] The original image 7, the watermark image 8, and the adversarial image 9 may have rectangular sizes. Then, for example, before converting the image into a tensor, the image size may be normalized to a square.

[0045] FIG. 1-2 is a flowchart showing an example of the flow of processing executed by the information processing device 1 in this embodiment. Note that the processes described below are merely examples of processes for realizing the method of the present disclosure, and are not limited to these. Furthermore, other steps may be added to the processing described below, or some steps may be omitted (deleted) from the processing described below.

[0046] First, the control unit (not shown) of the information processing device 1 executes an original image acquisition process (P110). In the original image acquisition process, for example, when the information processing device 1 acquires an arbitrary number of original images 7, it converts them into a tensor of the original images 7 "x".

[0047] Furthermore, the control unit of the information processing device 1 executes a watermark image acquisition process (P120). In the watermark image acquisition process, for example, when the information processing device 1 acquires the watermark image 8, it converts the watermark image 8 into a tensor of "m".

[0048] Here, the following image may be used as the watermark image. -Text image (e.g., name of the author or copyright holder of the original image 7) Logo images and signature images (for example, the signature of the author or copyright holder of the original image 7) Fingerprint image (e.g., fingerprint of the author or copyright holder of the original image 7) - coded images (e.g., barcodes that indicate an ID uniquely associated with the author or copyright holder of the original image7) These pieces of information may be called identification information that is included in the watermark image and is visible to the user.

[0049] The identification information included in the watermark image may be used to identify a user authorized to generate an image (based on the original image) that does not include a watermark image as a diffusion-generated image in the diffusion model unit 3. Here, the generation of an image based on the original image may involve (i) generating a diffusion-generated image based on the original image 7 without additional training of the diffusion model unit 3, i.e., inputting the original image 7 as a reference image into the diffusion model unit 3. Alternatively, (ii) generating a diffusion-generated image based on additional training of the diffusion model unit 3, i.e., using the original image 7 for additional training of the diffusion model unit 3 and inputting the reference image into the diffusion model unit 3 after additional training, may also be used to generate a diffusion-generated image. Alternatively, both (i) and (ii) may be included. In other words, the identification information included in the watermark image may be used to identify a user authorized to use the original image 7 as input for the diffusion model unit 3 or for additional training of the diffusion model unit 3.

[0050] In addition, if the sizes of the original image 7 and the watermark image 8 are different, the information processing device 1 may, for example, adjust (enlarge or reduce) the size of the original image 7 or the watermark image 8 to match the sizes of the original image 7 and the watermark image 8, and then convert them into tensors. Furthermore, if the number of channels of the original image 7 and the watermark image 8 differ, the information processing device 1 may, for example, convert the image to one with fewer channels (for example, convert from a color image to a grayscale image), adjust the number of channels, and then convert it into a tensor.

[0051] Here, "acquisition" of an image can include not only acquisition (input) of an image on the device itself, but also, for example, input of an image from another functional unit (e.g., a memory unit not shown) on the device itself (internal input), input of an image (external input) or reception (external reception) from a device other than the device itself (external device), etc.

[0052] Then, the control unit of the information processing device 1 executes a perturbation calculation process (P130). In the perturbation calculation process, for example, the perturbation control unit 2 concatenates "x" and "m" for each channel and accepts them as input. This allows the perturbation, which is the output of the perturbation control unit 2, to be calculated using "m" as a condition for the input "x." Then, the perturbation control unit 2 calculates the perturbation "G(x|m)" according to the model weight "G."

[0053] Then, the control unit of the information processing device 1 executes the hostile image generation process (P140). In the adversarial image generation process, for example, the adversarial image generator 4 adds the original image 7 “x” and the perturbation “G(x|m)” to generate the adversarial image 9 “x′” = “x” + “G(x|m)”.

[0054] Here, the perturbation "G(x|m)" is a change that has a large effect on the diffusion generated image "θ(x')" of the diffusion model unit 3 without having a large apparent effect on the original image 7 "x".

[0055] Therefore, the control unit of the information processing device 1 executes a differential loss calculation process (P150). In the differential loss calculation process, for example, the differential loss calculation unit 5 calculates the differential loss according to the following formula.

number

[0056] When the hostile image is generated, the control unit of the information processing device 1 executes a diffusion image generation process based on the hostile image (P160). In the diffusion image generation process, for example, the diffusion model unit 3 inputs the adversarial image 9 "x'" as a reference image and generates a diffusion generated image "θ(x')".

[0057] Then, the control unit of the information processing device 1 executes the adversarial loss calculation process (P170). In the adversarial loss calculation process, for example, the adversarial loss calculation unit 6 calculates the differential loss according to the following formula.

number

[0058] Then, the control unit of the information processing device 1 executes an optimization loss calculation process (P180). The optimization loss "L" is, for example, a loss used to optimize the model weight "G" in the perturbation control unit 2, and may be calculated according to the following formula.

number

[0059] Here, "α" and "β" are parameters for balancing the adversarial loss and the differential loss. For example, "α" and "β" may be greater than "0". Alternatively, "α+β=1" may be used. Alternatively, "α=1" and "β=10", or "α=10" and "β=1" may be used. When “α” is large, the adversarial loss term becomes dominant in the optimization loss. Therefore, by setting “α” larger than “β”, the success rate of attacks in the diffusion model unit 3 can be improved. Conversely, when “β” is large, the differential loss term dominates the optimization loss. Therefore, by setting “β” larger than “α”, we can reduce the noise in the adversarial image 9 “x´” (e.g., improve the S / N ratio). By adjusting the parameters “α” and “β”, it is possible to strike a balance between the magnitude of noise in the adversarial image 9 “x′” and the effectiveness of the attack on the diffusion model unit 3.

[0060] The parameters “α” and “β” can be said to be parameters for adjusting the visibility of the identification information included in the watermark image 8 in the adversarial image 9.

[0061] When the optimization loss is calculated, the control unit of the information processing device 1 determines, for example, whether the optimization loss is equal to or greater than a preset threshold value (P190). Note that the control unit of the information processing device 1 may also determine, for example, whether the optimization loss is greater than a preset threshold value.

[0062] If it is determined that the optimization loss is equal to or greater than the threshold (P190: YES), the control unit of the information processing device 1 executes a perturbation control unit model weight adjustment process (P200). In the perturbation control unit model weight adjustment process, for example, the perturbation control unit 2 adjusts the model weight "G" by backpropagation so as to minimize the optimization loss "L". Note that the perturbation control unit 2 may adjust the model weight "G" by an optimization method such as a genetic algorithm (GA).

[0063] After adjusting the model weight "G", for example, the control unit of the information processing device 1 returns the process to P130.

[0064] If it is determined that the optimization loss is smaller than the threshold (P190: NO), the control unit of the information processing device 1 executes an adversarial image output process (P210).

[0065] The control unit of the information processing device 1 may, for example, shift the processing to the adversarial image output processing (P210) when the number of times the optimization loss calculation processing is executed is equal to or exceeds a predetermined number. In this case, the control unit of the information processing device 1 may output an optimization failure notification indicating that the optimization of the model weight "G" of the perturbation control unit 2 has failed, and may interrupt the processing. Alternatively, the parameters "α" and "β" may be changed and reset, and the processing may be re-executed from P180.

[0066] In the adversarial image output process, for example, the control unit of the information processing device 1 converts the adversarial image 9 “x′” generated in the adversarial image generation process from a tensor into an image, and outputs it as an adversarial image for the diffusion model unit 3. Then, the control unit of the information processing device 1 ends the process.

[0067] In the adversarial image output process, for example, the control unit of the information processing device 1 may output the optimized model weight “G” of the perturbation control unit 2.

[0068] Here, "output" of an image can include not only displaying the image on the device itself (display output), but also, for example, outputting the image to another functional unit of the device itself (internal output), outputting the image to a device other than the device itself (external device) (external output) or transmitting the image (external transmission), etc.

[0069] The processing from P130 to P200 is processing for optimizing the model weight "G" of the perturbation control unit 2. Therefore, the processing from P130 to P200 may be collectively referred to as perturbation control unit optimization processing or perturbation control unit learning processing.

[0070] Furthermore, if the model weight "G" of the perturbation control unit 2 optimized for the original image 7 and the watermark image 8 has already been calculated, the control unit of the information processing device 1 may output the adversarial image 9 without performing the perturbation control unit optimization process. In this case, for example, the control unit of the information processing device 1 executes the original image acquisition process and the watermark image acquisition process to obtain the optimized model weight "G". Then, the control unit of the information processing device 1 executes the perturbation calculation process using the optimized model weight "G". This process may also be called a perturbation control unit inference process. Then, the control unit of the information processing device 1 may execute the adversarial image generation process based on the generated perturbation, and output the generated adversarial image 9 in the adversarial image output process. In this way, by executing the perturbation control unit inference process without performing the perturbation control unit optimization process, the adversarial image 9 based on the original image 7 and the watermark image 8 can be generated quickly.

[0071] <Example of generating diffuse generated images based on adversarial images> When the adversarial image in this embodiment is used in the diffusion model, for example, the methods of using the adversarial image can be roughly divided into the following. · Diffusion image generation method (A): When an adversarial image is used as the reference image. The diffusion model unit 3 generates a diffusion generated image based on, for example, an adversarial image and a prompt. Diffusion image generation method (B): When adversarial images are used as images for additional learning (fine tuning). The model weight “θ” of the diffusion model unit 3 is perturbed to the model weight “θ′” by additional learning using an adversarial image, but the diffusion generated image “θ′(x′)” is still close to the original image 7 “x”.

[0072] FIG. 1-3 shows three examples (A) to (C) of diffuse generated images generated based on an adversarial image in the diffuse generated image generation method (A). In Figures 1-3, the watermark image 8 uses the character string images "IMAGENET_CAT," "IMAGENET_DOG," and "IMAGENET_SHARK" from left to right. The perturbation control unit optimization process using 5-10 samples as the original image required 2-3 minutes of processing time when executed on an NVIDIA A100 80GB GPU. After the perturbation control unit optimization process, the adversarial image generation process required 0.2 seconds to generate one adversarial image.

[0073] In each example in Figures 1-3, from top to bottom, there are shown an original image 7, an adversarial image 9 generated using the original image 7 and a watermark image 8, a diffusion-generated image generated in the diffusion model unit 3 using the original image 7 as a reference image, and a diffusion-generated image generated in the diffusion model unit 3 using the adversarial image as a reference image. The prompt for generating the diffusion-generated image was "A painting." Each image was an RGB image of 512 x 512 pixels.

[0074] From these examples, we can see that the original image 7 and the adversarial image 9 are so similar that they are almost indistinguishable visually. We can also see that the watermark image 8 is generated in a visibly visible state in the diffusion generated image generated using the adversarial image 9 as a reference image. In the adversarial image 9, the visibility of the identification information contained in the watermark image 8 is low (or not visible), but in the diffusion generated image generated using the adversarial image 9 as a reference image, the visibility of the identification information contained in the watermark image 8 is significantly improved compared to the adversarial image 9. That is, the user can detect the identification information contained in the watermark image 8 by visually checking the diffused generated image.

[0075] Figure 1-4 shows an example of a generated image in the diffusion-generated image generation method (B), for example, when the diffusion model unit 3 is fine-tuned using an adversarial image 9 using the Textual Inversion method in the Stable Diffusion model. In Textual Inversion, for example, when an image for additional learning is given, the diffusion model unit 3 learns to express the concept of the given image with the word "S*." Then, a diffusion-generated image is generated using "S*," which indicates the acquired concept, as a prompt.

[0076] The left side of Figure 1-4 shows a diffusion generated image generated by using the prompt "An oil painting of S*" and the prompt "A photo of S* on a boat" in the diffusion model unit 3 after additional learning when the original image 7 is used as an image for additional learning. When the original image 7 is additionally learned by textual inversion, a diffusion generated image is generated with the original image 7 as the concept "S*".

[0077] The right side of Figure 1-4 shows a diffusion-generated image generated by the same prompt in the diffusion model unit 3 after additional training when an adversarial image 9 is used as an image for additional training. The watermark image 8 used to generate the adversarial image 9 is a character string image of "IMAGENET_CAT." It can be seen that the diffusion generated image generated using the diffusion model that has been additionally trained with the adversarial image 9 has the character string image of the watermark image 8 appearing in a visible state. That is, it can be seen that the adversarial images 9 generated by this method are effective not only when used as reference images but also when used as images for additional learning.

[0078] <Effects of the First Embodiment> According to this embodiment, by generating an adversarial image 9 based on an original image 7 and a watermark image 8 including identification information, when a diffusion-generated image is generated by inputting the adversarial image 9 as a reference image in a diffusion model, the identification information included in the watermark image 8 can be visually recognized in the diffusion-generated image. Furthermore, according to this embodiment, when the adversarial image 9 is used for additional learning of a diffusion model, and a reference image (the adversarial image 9 or an image different from the adversarial image 9) is input to the diffusion model after the additional learning is performed to generate a diffusion-generated image, the identification information included in the watermark image 8 can be visually recognized in the diffusion-generated image.

[0079] That is, it is possible to easily identify from the diffusion-generated image whether the diffusion-generated image was generated by a "user who does not have the authority to use the original image 7 as a reference image for the diffusion model" by inputting the adversarial image 9 into the diffusion model as a reference image for the diffusion model, or whether the diffusion-generated image was generated by a "user who does not have the authority to use the original image 7 for additional training of the diffusion model" by using the adversarial image 9 for additional training of the diffusion model and inputting the reference image into the diffusion model after the additional training. This makes it possible to prevent the adversarial image 9 from being used for purposes not intended by the author of the original image 7, etc.

[0080] <Second Example> The second embodiment is an embodiment in which an adversarial image 9 is generated using an original image 7 and a watermark image 8 provided (specified) by a user of a terminal 20 in a server 10 equipped with the information processing device 1 described in the first embodiment.

[0081] The contents described in the second embodiment are similarly applicable to any of the other embodiments and other modified examples.

[0082] In the following embodiments, a service for generating an adversarial image 9 will be referred to as an "adversarial image generation service" as an example. The adversarial image generation service may be usable together with an image generation service using a diffusion model (referred to as a "diffuse image generation service"). An application for realizing the adversarial image generation service will be referred to as an "adversarial image generation application."

[0083] In the following description, the user of terminal 20A communicating with server 10 will be referred to as "user AA," the user of terminal 20B as "user BB," the user of terminal 20C as "user CC," . . .

[0084] <System configuration> 2-1 is a diagram illustrating an example of a system configuration of a communication system 100 according to this embodiment. In the communication system 100, for example, a server 10 is connected to one or more terminals 20 (terminal 20A, terminal 20B, terminal 20C, ...) via a network 30.

[0085] The server 10 has a function of providing, for example, an adversarial image generation service to a terminal 20 owned by a user via a network 30. The server 10 can also be expressed as an adversarial image generation service server, etc. In this embodiment, as an example, the user of the server 10 is a company that provides an adversarial image generation service (an operator of the adversarial image generation service). Note that the number of servers 10 and the number of terminals 20 connected to the network 30 are not limited to those described above. In this embodiment, the "hostile image generation service" is a service provided by a business operator (server 10) such as a company that provides hostile image generation services, and may be provided to a user (user terminal 20), for example.

[0086] The terminal 20 (terminal 20A, terminal 20B, terminal 20C, etc.) may be any information processing terminal capable of implementing the functions described in each embodiment. Examples of the terminal 20 include a smartphone, a mobile phone (feature phone), a computer (including, but not limited to, a desktop, laptop, tablet, etc.), a media computer platform (including, but not limited to, a cable or satellite set-top box, digital video recorder, etc.), a handheld computer device (including, but not limited to, a PDA (personal digital assistant), email client, etc.), a wearable device (glasses-type device, watch-type device, etc.), a VR (Virtual Reality) terminal, a smart speaker (a device for voice recognition), or other types of computers or communication platforms. The terminal 20 may also be referred to as an information processing terminal.

[0087] For example, the configurations of terminal 20A, terminal 20B, and terminal 20C can be the same. Furthermore, as necessary, the terminal used by user X may be expressed as terminal 20X, and user information in a predetermined service associated with user X or terminal 20X may or may not be expressed as user information X. The user information is information of a user associated with an account used by the user in a predetermined service. The user information includes, but is not limited to, information associated with a user, such as the user's name, an icon image of the user, the user's age, the user's gender, the user's address, the user's hobbies and interests, and a user identifier, which is input by the user or assigned by the predetermined service, and may be any one of these, or a combination thereof, or may not be the same.

[0088] The network 30 serves to connect the devices constituting the communication system 100. In other words, the network 30 refers to a communication network that provides connection paths so that the above-mentioned various devices can be connected and send and receive data.

[0089] One or more portions of network 30 may or may not be a wired or wireless network. Network 30 may include, by way of example, an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), a portion of the Internet, a portion of the Public Switched Telephone Network (PSTN), a cellular network, integrated service digital networks (ISDN), wireless LAN, long term evolution (LTE), code division multiple access (CDMA), Bluetooth, satellite communications, etc., or a combination of two or more thereof. Network 30 may include one or more networks 30.

[0090] The server 10 (not limited to, but an example of a server, information processing device, or information management device) has a function of providing a predetermined service (in this embodiment, an adversarial image generation service) to the terminal 20, etc. The server 10 may be any information processing device that can realize the functions described in each embodiment. Examples of the server 10 include a server device, a computer (e.g., a desktop, laptop, tablet, etc.), a media computer platform (e.g., a cable or satellite set-top box, a digital video recorder), a handheld computer device (e.g., a PDA, an email client, etc.), or other types of computers or communication platforms. The server 10 may also be referred to as an information processing device. When there is no need to distinguish between the server 10 and the terminal 20, the server 10 and the terminal 20 may or may not each be referred to as an information processing device.

[0091] [Hardware (HW) configuration of each device] The hardware configuration of each device included in the communication system 100 will be described.

[0092] (1) Hardware configuration of the terminal FIG. 2-1 shows an example of the hardware configuration of the terminal 20. The terminal 20 includes, for example, a control unit 21 (CPU: central processing unit), a storage unit 28, a communication I / F 22 (interface), an input / output unit 23, a clock unit 29A, and a position calculation information detection unit 29B. The HW components of the terminal 20 are connected to each other, for example, via a bus B. It is not essential that the HW configuration of the terminal 20 includes all of the components. For example, the terminal 20 may or may not be configured such that individual components or multiple components are detachable.

[0093] The communication I / F 22 transmits and receives various data via the network 30. The communication may be performed either wired or wirelessly, and any communication protocol may be used as long as mutual communication is possible. The communication I / F 22 has a function of communicating with various devices such as the server 10 via the network 30. The communication I / F 22 transmits various data to various devices such as the server 10 in accordance with instructions from the control unit 21. The communication I / F 22 also receives various data transmitted from various devices such as the server 10 and transmits it to the control unit 21. The communication I / F 22 may also be simply referred to as a communication unit. When the communication I / F 22 is configured as a physically structured circuit, it may also be referred to as a communication circuit.

[0094] The input / output unit 23 includes a device for inputting various operations to the terminal 20, a device for outputting processing results processed by the terminal 20, etc. The input / output unit 23 may be an integrated input unit and an output unit, or may be separate input unit and output unit, or may not be so.

[0095] The input unit is realized by any one or a combination of all types of devices that can accept input from a user and transmit information related to the input to the control unit 21. Examples of the input unit include hardware keys such as a touch panel, a touch display, and a keyboard, a pointing device such as a mouse, a camera (for inputting operations via moving images), and a microphone (for inputting operations by voice).

[0096] The output unit is realized by any one or a combination of all types of devices that can output the processing results processed by the control unit 21. Examples of the output unit include a touch panel, a touch display, a speaker (audio output), a lens (for example, 3D (three dimensions) output or hologram output), a printer, etc.

[0097] Although this is merely an example, the input / output unit 23 includes a display unit 24, a sound input unit 25, a sound output unit 26, and an imaging unit 27, for example.

[0098] The display unit 24 is realized by any one of all types of devices or a combination thereof that can display according to the display data written to the frame buffer. Examples of the display unit 24 include a touch panel, a touch display, a monitor (e.g., a liquid crystal display or an organic electroluminescence display (OLED)), a head mounted display (HDM), projection mapping, a hologram, and a device that can display images, text information, etc. in air (which may or may not be a vacuum). Note that these display units 24 may or may not be capable of displaying display data in 3D.

[0099] The sound input unit 25 is used to input sound data (including voice data; the same applies below.) The sound input unit 25 includes a microphone and the like. The sound output unit 26 is used to output sound data and includes a speaker and the like. The imaging unit 27 is used to acquire image data (including still image data and moving image data; the same applies below.) The imaging unit 27 includes a camera and the like.

[0100] When the input / output unit 23 is a touch panel, the input / output unit 23 and the display unit 24 may be disposed opposite each other and have approximately the same size and shape.

[0101] The clock unit 29A is a built-in clock of the terminal 20, and outputs time information (timekeeping information). The clock unit 29A is configured to include, for example, a clock that uses a crystal oscillator. The clock unit 29A can also be expressed as, for example, a timekeeping unit or a time information detection unit.

[0102] The clock unit 29A may or may not have a clock that conforms to the NITZ (Network Identity and Time Zone) standard or the like.

[0103] The position calculation information detection unit 29B is a functional unit that detects (measures) information (hereinafter referred to as "position calculation information") necessary for the control unit 21 to calculate (measure) the position of its own terminal 20. The position calculation information detection unit 29B can also be expressed as a position calculation sensor unit, for example.

[0104] The position calculation information detection unit 29B includes, for example, a satellite positioning sensor (satellite positioning unit) which is a sensor or unit for calculating the position of the terminal 20 using a satellite positioning system such as GPS (Global Positioning System), an inertial measurement sensor (inertial measurement unit (IMU (Inertial Measurement Unit))) which is a sensor or unit for calculating the position of the terminal 20 using an inertial navigation system, a UWB positioning sensor (UWB positioning unit) which is a sensor or unit for calculating the position of the terminal 20 using UWB (Ultra Wide Band), and the like.

[0105] The satellite positioning unit includes, for example, an RF receiving circuit that converts RF (Radio Frequency) signals, including positioning satellite signals transmitted from positioning satellites and received by an antenna (not shown), into digital signals, and a baseband processing circuit that performs correlation calculation processing on the digital signals output from the RF receiving circuit to capture the positioning satellite signals, and outputs information such as satellite orbit data and time data extracted from the positioning satellite signals as information for position calculation.

[0106] The inertial measurement unit has an inertial sensor that is a sensor that detects information necessary for calculating the position of the terminal 20 by inertial navigation calculation. The inertial sensor includes, for example, a three-axis acceleration sensor and a three-axis gyro sensor, and outputs the acceleration detected by the acceleration sensor and the angular velocity detected by the gyro sensor as information for position calculation.

[0107] The UWB positioning unit includes, for example, an ultra-wideband RF (Radio Frequency) receiving circuit that converts an ultra-wideband RF signal, including an ultra-wideband pulse signal for positioning transmitted from a positioning beacon and received by an antenna not shown, into a digital signal, and a relative position calculation processing circuit that calculates the relative position between the terminal 20 and the positioning beacon based on the digital signal output from the ultra-wideband RF receiving circuit. For example, the UWB positioning unit may or may not cause the terminal 20 to function as a positioning beacon by transmitting an ultra-wideband RF signal including an ultra-wideband pulse signal for positioning from an antenna not shown.

[0108] For example, control unit 21 calculates the position of its own terminal 20 at regular intervals or specific intervals based on the position calculation information detected by position calculation information detection unit 29B. The terminal position is referred to as the "terminal position," and the calculated terminal position is referred to as the "calculated terminal position." Control unit 21 may, but need not, associate the calculated terminal position with the date and time when the calculated terminal position was calculated and store the calculated terminal position in storage unit 28 as calculated terminal position history data.

[0109] The control unit 21 has a circuit physically structured to execute the functions realized by the code or instructions contained in the program, and is realized by, for example, a data processing device built into hardware. Therefore, the control unit 21 may or may not be expressed as a control circuit.

[0110] The control unit 21 includes, for example, a central processing unit (CPU), a microprocessor, a processor core, a multiprocessor, an application-specific integrated circuit (ASIC), and a field programmable gate array (FPGA).

[0111] The storage unit 28 has a function of storing various programs and various data required for the operation of the terminal 20. The storage unit 28 includes, for example, various storage media such as a hard disk drive (HDD), a solid state drive (SSD), a flash memory, a random access memory (RAM), and a read only memory (ROM). Furthermore, the storage unit 28 may or may not be expressed as a memory.

[0112] Terminal 20 stores program P in storage unit 28, and by executing this program P, control unit 21 executes the processing of each unit included in control unit 21. In other words, program P stored in storage unit 28 causes terminal 20 to realize each function executed by control unit 21. Furthermore, this program P may or may not be expressed as a program module.

[0113] (2) Server hardware configuration FIG. 2-1 shows an example of the hardware configuration of the server 10. In FIG. The server 10 includes, for example, a control unit 11 (CPU), a memory unit 15, a communication I / F 14 (interface), an input / output unit 12, a display unit 13, and a clock unit 19. The components of the HW of the server 10 are connected to each other, for example, via a bus B. Note that the HW of the server 10 does not necessarily have to include all components as the configuration of the HW of the server 10. For example, the HW of the server 10 may or may not be configured so that individual components or multiple components can be removed.

[0114] The control unit 11 has circuits that are physically structured to execute the functions realized by the codes or instructions contained in the program, and is realized, for example, by a data processing device built into hardware.

[0115] The control unit 11 is typically a central processing unit (CPU), but may also be a microprocessor, a processor core, a multiprocessor, an ASIC, or an FPGA. In the present disclosure, the control unit 11 is not limited to these.

[0116] The storage unit 15 has a function of storing various programs and various data required for the operation of the server 10. The storage unit 15 is realized by various storage media such as an HDD, an SSD, and a flash memory. However, in the present disclosure, the storage unit 15 is not limited to these. Furthermore, the storage unit 15 may or may not be expressed as a memory.

[0117] The communication I / F 14 transmits and receives various data via the network 30. The communication may be performed either wired or wirelessly, and any communication protocol may be used as long as mutual communication is possible. The communication I / F 14 has a function of communicating with various devices such as the terminal 20 via the network 30. The communication I / F 14 transmits various data to various devices such as the terminal 20 in accordance with instructions from the control unit 11. The communication I / F 14 also receives various data transmitted from various devices such as the terminal 20 and transmits it to the control unit 11. The communication I / F 14 may also be simply referred to as a communication unit. When the communication I / F 14 is configured as a physically structured circuit, it may also be referred to as a communication circuit.

[0118] The input / output unit 12 includes a device for inputting various operations to the server 10, a device for outputting processing results processed by the server 10, etc. The input / output unit 12 may be an integrated input unit and an output unit, or may be separate input unit and output unit, or may not be the same.

[0119] The input unit is realized by any one of or a combination of all types of devices that can accept input from a user and transmit information related to the input to the control unit 11. The input unit is typically realized by hardware keys such as a keyboard or a pointing device such as a mouse. Note that the input unit may or may not include, for example, a touch panel, a camera (for operation input via moving images), or a microphone (for operation input by voice).

[0120] The output unit is realized by any one or a combination of all kinds of devices that can output the processing results processed by the control unit 11. Examples of the output unit include a touch panel, a touch display, a speaker (sound output), a lens (for example, 3D (three dimensions) output or hologram output), a printer, etc.

[0121] By way of example only, the input / output unit 12 includes a display unit 13, for example.

[0122] The display unit 13 is realized by a display or the like. The display is typically realized by a monitor (for example, a liquid crystal display or an OLED (organic electroluminescence display)). The display may or may not be a head-mounted display (HDM) or the like. These displays may or may not be capable of displaying display data in 3D. In the present disclosure, the display is not limited to these.

[0123] The clock unit 19 is a built-in clock of the server 10, and outputs time information (timekeeping information). The clock unit 19 is configured to include, for example, an RTC (Real Time Clock) as a hardware clock, a system clock, etc. The clock unit 19 can also be expressed as, for example, a timekeeping unit or a time information detection unit.

[0124] (3) Other Server 10 stores program P in storage unit 15, and by executing this program P, control unit 11 executes the processes of each unit included in control unit 11. In other words, program P stored in storage unit 15 causes server 10 to realize each function executed by control unit 11. This program P may or may not be expressed as a program module. The same applies to other devices.

[0125] In each embodiment of the present disclosure, the description will be given assuming that the CPU of the terminal 20 and / or the server 10 executes the program P to realize the present invention. The same applies to other devices.

[0126] The control unit 21 of the terminal 20 and / or the control unit 11 of the server 10 may or may not realize each process not only by a CPU having a control circuit but also by a logic circuit (hardware) formed in an integrated circuit (IC (Integrated Circuit) chip, LSI (Large Scale Integration)), or a dedicated circuit. These circuits may be realized by one or more integrated circuits, and multiple processes shown in each embodiment may or may not be realized by a single integrated circuit. LSIs may also be called VLSIs, super LSIs, ultra LSIs, etc. depending on the degree of integration. Therefore, the control unit 21 may or may not be expressed as a control circuit. The same applies to other devices.

[0127] Furthermore, the program P (e.g., a software program, a computer program, or a program module) of each embodiment of the present disclosure may or may not be provided in a state stored in a computer-readable storage medium. The storage medium can store the program P in a "non-transitory tangible medium." The program P may or may not be intended to realize part of the functions of each embodiment of the present disclosure. Furthermore, the program P may or may not be a so-called differential file (differential program) that can realize the functions of each embodiment of the present disclosure in combination with a program P already recorded on a storage medium.

[0128] The storage medium may include one or more semiconductor-based or other integrated circuits (ICs) (such as, for example, field programmable gate arrays (FPGAs) or application-specific ICs (ASICs)), hard disk drives (HDDs), hybrid hard drives (HHDs), optical disks, optical disk drives (ODDs), magneto-optical disks, magneto-optical drives, floppy diskettes, floppy disk drives (FDDs), magnetic tapes, solid-state drives (SSDs), RAM drives, secure digital cards, or drives, any other suitable storage media, or any suitable combination of two or more of these. The storage medium may be volatile, nonvolatile, or a combination of volatile and nonvolatile, where appropriate. The storage medium is not limited to these examples and may be any device or medium capable of storing the program P. Furthermore, the storage medium may or may not be referred to as memory.

[0129] The server 10 and / or the terminal 20 can implement the functions of the multiple functional units shown in each embodiment by reading out the program P stored in the storage medium and executing the read out program P. The same applies to other devices.

[0130] Furthermore, the program P of the present disclosure may or may not be provided to the server 10 and / or the terminal 20 via any transmission medium capable of transmitting a program (such as a communication network or broadcast waves). The server 10 and / or the terminal 20 executes the program P downloaded via the Internet or the like, for example, to realize the functions of the multiple functional units shown in each embodiment. The same applies to other devices.

[0131] In addition, each embodiment of the present disclosure may also be realized in the form of a data signal in which the program P is embodied by electronic transmission. At least a part of the processing in the server 10 and / or the terminal 20 may or may not be realized by cloud computing configured by one or more computers. At least a part or all of the processing in the terminal 20 may or may not be performed by the server 10. In this case, at least a part or all of the processing of each functional unit of the control unit 21 of the terminal 20 may or may not be performed by the server 10. At least a part or all of the processing in the server 10 may or may not be performed by the terminal 20. In this case, at least a part or all of the processing of each functional unit of the control unit 11 of the server 10 may or may not be performed by the terminal 20. Unless explicitly stated otherwise, the judgment configuration in the embodiments of the present disclosure is not essential, and a predetermined process may or may not be performed when the judgment condition is met, or when the judgment condition is not met.

[0132] The program of the present disclosure is implemented using, for example, a scripting language such as ActionScript or JavaScript (registered trademark), a compiler language such as Objective-C or Java (registered trademark), or a markup language such as HTML Living Standard.

[0133] [Functional configuration of each device] (1) Server functional configuration FIG. 2-2 is a diagram showing an example of functions realized by the control unit 11 of the server 10 in this embodiment. The control unit 11 includes, for example, an adversarial image generation unit 1 and an application management processing unit 111 as functional units.

[0134] The adversarial image generation unit 1 includes, for example, the functional units shown in FIG. 1-1. The application management processing unit 111 also has a function of executing application management processing in accordance with an application management processing program 151 stored in the storage unit 15, for example.

[0135] FIG. 2-3 is a diagram showing an example of information stored in the storage unit 15 of the server 10 in this embodiment. The storage unit 15 stores, for example, an application management processing program 151 that is executed as application management processing, and account registration data 153.

[0136] The account registration data 153 is registration data related to the account of the application (in this embodiment, the adversarial image generation application), and an example of the data configuration is shown in FIG. 2-4. The account registration data 153 stores, for example, a user name, an application ID, watermark image information, and other registration information in association with each other.

[0137] The user name is the name of the account of the terminal 20 that uses this application, and for example, the name that the user of the terminal 20 registers when using the application is stored.

[0138] The application ID is information used to identify an application account, or the account itself. This application ID is preferably a value that is unique for each account, and for example, a unique value (proper value) is set and stored by the server 10 for each account. The application ID is information associated with the terminal 20 or the user of the terminal 20, and is an example of information about the terminal or information about the user of the terminal.

[0139] The watermark image information is a watermark image 8 associated with a user. The watermark image information can be specified by the user and stored in the server 10, for example. The watermark image information may be referred to as image author identification information or copyright information, or as image owner identification information or ownership information.

[0140] Note that a plurality of watermark images 8 may be associated with one application ID and stored as watermark image information. That is, a watermark image 8 may be prepared for each original image 7 (original image set), and a plurality of watermark images 8 for the plurality of original images 7 may be used to generate respective adversarial images 9.

[0141] Other registration information may include, for example, various types of information such as identification information for identifying terminal 20, the telephone number of terminal 20 (terminal telephone number), email address (terminal email address), and authentication information such as passwords (login password, authentication password, etc.) used for various authentications in applications.

[0142] The identification information for identifying the terminal 20 may be, for example, a terminal ID (for example, an IMEI (International Mobile Equipment Identity)).

[0143] Note that the application ID may or may not be replaced with a "user ID." Furthermore, if the application allows only one account to be registered per terminal 20, then for example, "identification information for identifying the terminal 20 = identification information for identifying the user of the terminal 20 = application ID."

[0144] Also, for example, it may or may not be possible to assign multiple terminal IDs to one application ID. In this case, it may or may not be possible to simultaneously launch an application on multiple terminals 20 using one application ID as an identification (login) target.

[0145] Furthermore, instead of various IDs such as application IDs, it is also possible to apply a method of managing accounts using information such as terminal phone numbers. In this case, instead of storing information on IDs such as application IDs in account registration data 153, information on terminal telephone numbers and the like can be stored in account registration data 153. It is also possible to have one-to-one correspondence between information on IDs such as application IDs and information on terminal telephone numbers and the like, without replacing information on IDs such as application IDs with information on terminal telephone numbers and the like, but this is not required.

[0146] In the following embodiment, for the sake of simplicity, it is assumed that one account is registered for one terminal 20. Furthermore, in this case, as described above, "identification information for identifying terminal 20 = identification information for identifying the user of terminal 20 = application ID," so the term "user of the account" used in the following explanation may or may not be substantially synonymous with "terminal of the account."

[0147] (2) Functional configuration of the terminal FIG. 2-5 is a diagram showing an example of functions realized by the control unit 21 of the terminal 20 in this embodiment. The control unit 21 includes, as a functional unit, an application processing unit 211 for executing application processing in accordance with an application processing program 281 stored in the storage unit 28, for example.

[0148] FIG. 2-6 is a diagram showing an example of data stored in the storage unit 28 of the terminal 20 in this embodiment. The storage unit 28 stores, for example, an application processing program 281 to be executed as application processing, and an application ID 283 corresponding to the terminal 20 or the account of the user of the terminal 20.

[0149] <Display screen> Examples of display screens will be described below. The transitions of the display screens described below are merely examples of transitions of the display screens for realizing the technique of the present disclosure. In the transitions of the display screens exemplified below, the display of some of the display screens may be omitted, or other display screens may be added. Furthermore, the terms used on the display screens described below may differ from the terms used in the processes described thereafter (some terms may not be consistent).

[0150] In the following, the terminal 20 is exemplified as a smartphone having a display unit 24 with a vertically long display. In the smartphone, a touch panel functioning as an input unit is disposed opposite the display, forming a touch screen. When an element such as an icon, button, item, or input area is displayed on the display, and a part of the touch panel that faces the area where the element is displayed is operated by a user, a program associated with the element or a subroutine of the program may be executed.

[0151] Note that the flow of screen transitions may be explained by showing specific screens displayed on different terminals 20 within one drawing or across multiple drawings. In this case, for example, the specific screen may be displayed on the different terminals 20 based on (triggered by) at least one of the following: - Automatically displayed A notification (including a push notification, etc.; the same applies below) is sent to the terminal 20, and is displayed when the user performs an operation in response to the notification. No notification is sent to the terminal 20, but it is displayed when the user performs a predetermined operation on the application. A notification is sent to the terminal 20, and the message is displayed when the user performs a predetermined operation on the application that sent the notification.

[0152] In the following, an example of an account that generates an adversarial image is assumed to be terminal 20A of user AA. Terminal 20A may be an example of a first terminal, and user AA may be an example of a first user of the first terminal. Furthermore, terminal 20A or the account of user AA may be an example of a first account. In this example, the generation of the adversarial image is performed within an adversarial image generation application, and the name of the adversarial image generation application is shown as "Image Protection App." The generation of the adversarial image may be executed on, for example, a messaging application different from the adversarial image generation application. In this case, for example, the original image and the watermark image may be acquired based on a conversation in a chat room of the messaging application, and the adversarial image may be presented.

[0153] FIG. 2-7 is a diagram showing an example of a screen displayed on the display unit 24 of the terminal 20A in this embodiment. In the drawings, what is referred to as a "base image" will be referred to as an "original image" where appropriate. Also, what is referred to as a "digital watermark" will be referred to as a "watermark image" where appropriate. Also, what is referred to as a "protection image" will be referred to as an "adversarial image" where appropriate.

[0154] The left side of Fig. 2-7 shows an example of an original image selection screen of the adversarial image generation application displayed on the display unit 24 of the terminal 20A. On this screen, for example, four images are selected as original images from images stored in an image database (not shown) stored in the storage unit 28 of the terminal 20A.

[0155] When the original image is selected, the display changes to, for example, a watermark image selection screen in the center. On the watermark image selection screen, for example, an image to be applied to the image selected as the original image (in this screen, an image including the character string "Genuine AA" as identification information) is selected from among the watermark images created in advance by the user of terminal 20. Note that, for example, it may be possible to create a new watermark image on this screen. Note that selecting a watermark image may also be considered as selecting (or inputting or setting) the identification information included in the watermark image.

[0156] When a watermark image is selected, for example, the selected original image and the watermark image are transmitted to the server 10, and the perturbation control unit optimization process and the adversarial image output process are executed in the server 10. Then, the generated adversarial image is transmitted to the terminal 20A.

[0157] Then, for example, the display changes to the hostile image confirmation screen on the right. This screen displays a list of the generated hostile images. This screen also displays a "Save Selected Image" button for saving an image selected by the user from the hostile images to the terminal 20, and a "Save All Images" button for saving all hostile images at once.

[0158] In addition, on the hostile image confirmation screen, the hostile image may be made available for sharing (posting) to any SNS service, messaging service, image sharing service, or the like.

[0159] <Processing> 2-8 is a flowchart showing an example of the flow of processing executed by each device in this embodiment. From the left, this figure shows an example of processing executed by the control unit 21 of the terminal 20A of user AA, and an example of processing executed by the control unit 11 of the server 10.

[0160] First, the control unit 21 of the terminal 20A selects and acquires original image information including one or more original images based on, for example, an input to the input / output unit 23 of the terminal 20A (for example, a user input (such as an operation input or sound input by the user) and the same applies hereinafter). Note that the original images may be captured by, for example, the imaging unit 27 of the terminal 20A. Then, the control unit 21 of the terminal 20A transmits the original image information to the server 10 via the communication I / F 22 (the same applies below) (A110).

[0161] Furthermore, when the control unit 21 of the terminal 20A selects and acquires watermark image information including a watermark image based on, for example, a user input, it transmits the watermark image information to the server 10 (A120).

[0162] When the original image information and watermark image information are received from the terminal 20A via the communication I / F 14 (similarly hereinafter), the account registration data 153 is referenced, and the received watermark image information is added and stored in association with the application ID of the terminal 20A. Then, the control unit 11 of the server 10 executes the perturbation control unit optimization process (S110). The perturbation control unit optimization process is executed, for example, in accordance with steps P130 to P200 in FIG. 1-2.

[0163] In addition, the control unit 11 of the server 10 may refer to the account registration data 153, and if watermark image information identical to or similar to the received watermark image information (for example, the correlation between images is higher than a threshold) is already registered, send watermark image retransmission request information to the terminal 20A to request the retransmission of watermark image information including a different watermark image. When receiving watermark image resend request information from the server 10, the control unit 21 of the terminal 20A may output (e.g., display) the received watermark image resend request information and, based on user input, may transmit the watermark image information to the server 10 again.

[0164] Then, the control unit 11 of the server 10 executes the adversarial image generation process (S120). The adversarial image generation process is executed, for example, according to step P140 in FIG. 1-2 using the optimized (post-learning) model weight “G” of the perturbation control unit 2.

[0165] Then, the control unit 11 of the server 10 transmits hostile image information including the generated hostile image to the terminal 20A (S130).

[0166] The control unit 11 of the server 10 may store, for example, the model weight "G" of the perturbation control unit 2 optimized in the perturbation control unit optimization process, the original image information, and the watermark image information in association with each other in the account registration data 153. Then, optimized model weight list information, which is a list of the stored model weights of the perturbation control unit 2, may be transmitted to the terminal 20. The optimized model weight list information may include the original image and the watermark image used in the perturbation control unit optimization process.

[0167] Upon receiving the optimized model weight list information, the control unit 21 of the terminal 20 may, for example, display the list information. Then, the control unit 21 may transmit, to the server 10, model weight designation information for designating model weights to be used for generating adversarial images, out of the optimized model weight list information. The control unit 11 of the server 10 may execute an adversarial image generation process using the model weight “G” specified by the model weight specification information, the original image information, and the watermark image information, and transmit adversarial image information including the generated adversarial image to the terminal 20. This may allow an adversarial image to be generated without performing the perturbation control unit optimization process again when using optimized original image information and watermark image information.

[0168] The control unit 11 of the server 10 determines whether to end the process (S190). If it is determined that the process should be continued (S190: NO), the control unit 11 of the server 10 waits for reception of original image information, for example, and returns the process to S110. On the other hand, if it is determined that the process should be ended (S190: YES), the control unit 11 of the server 10 ends the process.

[0169] When the hostile image information is received from the server 10, the control unit 21 of the terminal 20A outputs the received hostile image information (for example, displays it on the display unit 24). The control unit 21 of the terminal 20A may store the received hostile image information in the storage unit .

[0170] This allows the user of terminal 20A to easily obtain an adversarial image that is comparable in appearance to the original image, but that reveals the watermark image when used as a reference image by the diffusion model. Furthermore, the perturbation control unit optimization process may be difficult to perform on a portable terminal such as a smartphone due to issues with the processor's processing power (e.g., the number of parallel processes), resources (e.g., the amount of memory used for calculations), power consumption, etc. However, by executing the perturbation control unit optimization process on the server 10, which has ample processing power and allowable power consumption, the perturbation control unit optimization process can be realized within a practical processing time, regardless of the processing power, etc. of the terminal 20.

[0171] Then, the control unit 21 of the terminal 20A determines whether or not to end the processing (A190). If it is determined to continue the processing (A190: NO), the control unit 21 of the terminal 20A returns the processing to A110, for example. On the other hand, if it is determined to end the processing (A190: YES), the control unit 21 of the terminal 20A ends the processing.

[0172] <Effects of the second embodiment> According to this embodiment, an author or the like uses his / her terminal 20 to select an original image that he / she wishes to protect and a watermark image that he / she wishes to use, and an adversarial image based on the selected original image and watermark image is generated by the server 10 and provided to the terminal 20. The author or the like can easily obtain an adversarial image based on the original image that he / she wishes to protect. Furthermore, as will be described in the following embodiment, by registering the adversarial image thus obtained in the server 10, it is possible to prevent the adversarial image based on the original image from being used as a reference image for the diffusion model or for additional training of the diffusion model.

[0173] <Second Modification Example (1)> In the above embodiment, the perturbation control unit optimization process, the adversarial image generation process, and communication with the terminal 20 are performed in the server 10, but the present invention is not limited to this. For example, the server 10 may be a server system including a front-end server 10F that provides a messaging service and an image transmission / reception interface service, and a back-end server 10B that provides perturbation control unit optimization processing and adversarial image generation processing.

[0174] The process in this case will be exemplified below. For example, the front-end server 10F receives the original image information and the watermark image information from the terminal 20. Then, the front-end server 10F refers to the account registration data 153 and registers the watermark image information.

[0175] Then, the front-end server 10F transmits the adversarial image generation request information including the original image information and the watermark image information to the back-end server 10B. The back-end server 10B executes the perturbation control unit optimization process and the adversarial image generation process based on the received adversarial image generation request information, and then transmits the adversarial image information including the generated adversarial image to the front-end server 10F.

[0176] Upon receiving the hostile image information, the front-end server 10F transmits it to the terminal 20.

[0177] <Second Modification Example (2)> In the above embodiment, the server 10 transmits an adversarial image to the terminal 20 when the perturbation control unit optimization process is executed, but the present invention is not limited to this. For example, when the server 10 executes the perturbation control unit optimization process, the server 10 may transmit perturbation control unit model weight information including the model weight “G” of the optimized perturbation control unit 2 to the terminal 20 .

[0178] Upon receiving the perturbation control unit model weight information, the terminal 20 may store, for example, the optimized model weight “G”, the original image information, and the watermark image information in the storage unit 28.

[0179] Then, when generating an adversarial image based on the same original image information and watermark image information again, the model weight “G” may be transmitted to the server 10 in addition to the original image information and the watermark image information. When the server 10 receives the model weight "G" from the terminal 20, the server 10 may omit the perturbation control unit optimization process and execute the adversarial image generation process based on the received model weight "G".

[0180] Note that, upon receiving the optimized model weight “G” from the server 10, the terminal 20 may execute the adversarial image generation process. The adversarial image generation process does not require the operation of the diffusion model unit 3. Therefore, by implementing only the perturbation control unit 2, which has a relatively simple structure compared to the diffusion model unit 3, in the terminal 20 and executing the adversarial image generation process, the load on the server 10 can be reduced. Furthermore, the user of the terminal 20 can generate and acquire adversarial images at any time, regardless of the operating status or communication conditions of the server 10.

[0181] <Third Example> In the third embodiment, for example, when a second user generates a diffusion-generated image using an adversarial image generated by a first user as a reference image, the server 10 notifies the second user of information about the first user who generated the adversarial image (e.g., the author of the original image) based on a watermark image embedded in the adversarial image.

[0182] The contents described in the third embodiment can be similarly applied to any of the other embodiments and other modified examples.

[0183] <Display screen> In this example, a service that generates a diffusion image based on a diffusion model using a reference image is referred to as a diffusion image generation service, and an application for realizing the diffusion image generation service is referred to as a "diffusion image generation application." The name of the diffusion image generation application is exemplified as "Image Creation App." The diffusion image generation application may allow any user to register a reference image.

[0184] In the following, a case will be exemplified in which user AA's terminal 20A is an example of an account that registers an adversarial image generated in the adversarial image generation service as an example of a reference image in the diffusion image generation service, and user BB's terminal 20B is an example of an account that generates a diffusion-generated image using the adversarial image as a reference image in the diffusion image generation service.

[0185] The diffusion image generation service and the adversarial image generation service may be integrated or may be separate services. The diffusion models used in the diffusion image generation service and the adversarial image generation service may have common model weights. Furthermore, the diffusion model used in the diffusion image generation service may be a diffusion model that has undergone fine tuning or additional learning, with the diffusion model used in the adversarial image generation service being used as a pre-trained model. In this example, for example, the diffuse image generation service and the adversarial image generation service are separate services (different applications), and the accounts for the respective services are linked.

[0186] 3-1 and 3-2 are diagrams showing examples of screens displayed on the display unit 24 of each terminal 20 in this embodiment. In the drawings, what is referred to as a "reference image" will be referred to as a "reference image" where appropriate. Also, what is referred to as a "protected image" will be referred to as a "hostile image" where appropriate.

[0187] The left side of FIG. 3-1 shows an example of a reference selection screen of the diffusion image generation application displayed on the display unit 24 of the terminal 20B. On this screen, for example, one image is selected as a reference image from images stored in a reference image database (not shown) stored in the storage unit 28 of the server 10. The reference image database may contain adversarial images. In this example, the selected reference image is the adversarial image generated in FIG. 2-7.

[0188] The reference image may be selected from images stored in an image database (not shown) stored in the storage unit 28 of the terminal 20B. The image database may store images acquired by the user BB from an SNS service, an image sharing service, or the like. In this case, the image database of the terminal 20B may include hostile images.

[0189] When a reference image is selected, the display changes to, for example, a prompt input screen on the right. On the prompt input screen, when an image generation prompt is input in the prompt input region PTR and the "Generate" button at the bottom of the screen is tapped, a diffusion generated image generation process is executed, for example, in the diffusion model unit 3 of the server 10. Then, a diffusion generated image is generated based on the reference image and the input prompt. The generated diffusion generated image is transmitted to the terminal 20B.

[0190] Then, for example, the display changes to the diffuse generated image confirmation screen shown on the left side of Figure 3-2. On this screen, the generated diffuse generated image contains the string "Genuine AA" (appears as a watermark). This allows user BB, who generated the diffuse generated image, to realize that he or she used an adversarial image as a reference image.

[0191] Additionally, below the diffusely generated image, hostile image use warning information is displayed to notify the user that a hostile image has been used as a reference image. The hostile image use warning information displays, for example, information about user AA, who generated and registered the hostile image. This allows user BB, who generated the diffusely generated image, to be notified that the contact person for image usage permission, etc. is user AA.

[0192] Also, for example, if an adversarial image is used as a reference image, a notification may be sent to the adversarial image generation application. On the right side, an example of output of warning information about the use of an adversarial image in the adversarial image generation application is shown. This screen displays information about user BB, who attempted to generate a diffusion-generated image using a hostile image as a reference image. Note that the hostile image use warning information may include, for example, the hostile image used as a reference image.

[0193] <Processing> 3-3 is a flowchart showing an example of the flow of processing executed by each device in this embodiment. From the left, this figure shows an example of processing executed by the control unit 21 of the terminal 20A of user AA, an example of processing executed by the control unit 11 of the server 10, and an example of processing executed by the control unit 21 of the terminal 20B of user BB. In this example, the server 10 may provide an adversarial image generation service and a diffusion image generation service.

[0194] First, the control unit 21 of the terminal 20A transmits, for example, based on a user input, to the server 10 hostile image registration information for enabling the hostile image generated in the hostile image generation service to be used as a reference image in the diffusion image generation service (A210). The hostile image registration information may include one or more hostile images.

[0195] When receiving the hostile image registration information from the terminal 20A, the control unit 11 of the server 10 executes a hostile image registration process (S210). In the hostile image registration process, the control unit 11 of the server 10 stores a hostile image in a reference image database based on the received hostile image registration information, for example.

[0196] The control unit 21 of the terminal 20B transmits reference image list request information for acquiring a list of images to be used as reference images in the spread image generation service to the server 10 based on, for example, a user input (B210). Then, the control unit 11 of the server 10 refers to the reference image database and transmits reference image list information, which is a list of registered reference images, to the terminal 20B (S220).

[0197] For example, when an image to be used as a reference image and a prompt are received from the reference image list information based on a user input, the control unit 21 of the terminal 20B transmits diffuse generated image generation request information including the reference image and the prompt to the server 10 (B220). The diffused image generation request information may not include a prompt. The reference image may also be an image stored in the storage unit 28 of the terminal 20B, for example.

[0198] When receiving the diffusion generated image generation request information from the terminal 20B, the control unit 11 of the server 10 executes a diffusion generated image generation process (S230). In the diffusion generated image generation process, the control unit 11 of the server 10 inputs a reference image and a prompt to the diffusion model unit 3, for example, to generate a diffusion generated image. Note that in the diffusion generated image generation process, the control unit 11 of the server 10 may input a reference image to the diffusion model unit 3 to generate a diffusion generated image.

[0199] Then, the control unit 11 of the server 10 transmits the diffused generated image information including the generated diffused generated image to the terminal 20B (S240).

[0200] When receiving the diffusion generated image information from the server 10, the control unit 21 of the terminal 20B outputs (for example, displays) the received diffusion generated image information (B230). Note that the control unit 21 of the terminal 20B may store the diffusion generated image included in the received diffusion generated image information in the storage unit 28.

[0201] Then, the control unit 11 of the server 10 executes a watermark image information detection process (S250). In the watermark image information detection process, the control unit 11 of the server 10, for example, refers to the account registration data 153 and determines whether or not the diffusion generated image includes each image registered as watermark image information. The determination of whether or not watermark image information is included may be made based on, for example, the correlation between images. Alternatively, the determination of whether or not watermark image information is included may be made using, for example, an API provided by a watermark detection server (not shown).

[0202] If it is determined that a watermark image has been detected (S250: YES), the control unit 11 of the server 10 transmits, for example, hostile image use warning information indicating that a hostile image has been used as a reference image to the terminal 20A and the terminal 20B (S260). When determining that a watermark image is not detected (S250: NO), the control unit 11 of the server 10, for example, ends the process.

[0203] When it is determined that hostile image use warning information has been received from the server 10 (A220: YES), the control unit 21 of the terminal 20A, for example, outputs (for example, displays) the received hostile image use warning information (A230). When it is determined that the hostile image use warning information has not been received from the server 10 (A220: NO), the control unit 21 of the terminal 20A, for example, ends the process.

[0204] The same applies to the terminal 20B.

[0205] In the system disclosed in this specification, even if it is determined whether a watermark image is detected in a diffusion-generated image after the diffusion-generated image is transmitted to the creator's terminal 20, no problems (such as copyright infringement) will occur. This is because the watermark image is embedded in the diffusion-generated image itself in a state that is visible to the user.

[0206] <Effects of the third embodiment> According to this embodiment, when an adversarial image is used as a reference image for a diffusion model to generate a diffusion-generated image, or when an adversarial image is used for additional learning of a diffusion model and a diffusion-generated image is generated using the diffusion model after the additional learning, warning information is sent to the terminal of the user who is the author of the original image or the terminal of the user who attempted to generate the diffusion-generated image, thereby preventing the adversarial image from being used for purposes not intended by the author of the original image, etc.

[0207] <Third Modification (1)> In the above embodiment, the adversarial image generation service and the diffuse image generation service are provided by the server 10, but the present invention is not limited to this. For example, a server 10A that provides an adversarial image generation service and a server 10D that provides a diffuse image generation service may be separated.

[0208] For example, the server 10A provides the terminal 20 with an adversarial image based on the original image and the watermark image according to FIGS. 2-8. Also, for example, the server 10D provides the terminal 20 with a diffusion-generated image based on the adversarial image according to FIG. 3-3. The server 10D may have a diffusion model configured with a model weight “θ” similar to that of the diffusion model unit 3 in the server 10A, or a diffusion model configured with a model weight “θ′” obtained by fine-tuning or additional learning the diffusion model unit 3. The server 10D does not require the perturbation control unit 2.

[0209] 3-3, for example, the terminal 20A may transmit the hostile image and the watermark image to the server 10D. Then, the server 10D may store the watermark image information in the storage unit 15 in association with the account information of the terminal 20A.

[0210] In the information processing device 1 disclosed in this specification, it is not necessary to change (learn) the model weight "θ" of the diffusion model unit 3 in order to generate an adversarial image. This allows the servers 10 that provide services to be flexibly separated.

[0211] <Third Modification (2)> In the above embodiment, the adversarial image is generated based on the above-mentioned diffusion-generated image generation method (A), but this is not limiting. For example, the adversarial image may be generated based on the diffusion-generated image generation method (B) for additional learning. In this case, for example, if a prompt associated with the style of an adversarial image is used in a diffusion model that has learned the style of an adversarial image, a watermark image is embedded in the diffusion-generated image. Therefore, for example, if an image similar to the style of an adversarial image is generated in the learned diffusion model, the server 10 can transmit adversarial image use warning information to the terminal 20.

[0212] <Third Modification (3)> In the above embodiment, when a watermark image is detected in a diffusion-generated image, hostile image use warning information is sent to terminal 20A, which generated a hostile image and made it usable as a reference image, and terminal 20B, which generated a diffusion-generated image using the hostile image as a reference image, but this is not limited to this.

[0213] Figure 3-4 shows a list of various patterns that can be set as destinations for warning information about the use of hostile images. In pattern (A), for example, an adversarial image is sent to a user who has registered it as a reference image in a diffusion image generation service. In pattern (B), for example, the adversarial image is sent to a user who generated an adversarial image from an original image in an adversarial image generation service. In pattern (C), for example, a diffuse image generated using an adversarial image as a reference image in a diffuse image generation service is sent to a user who has generated the diffuse image. In pattern (D), for example, the data is sent to a user who has performed additional learning of a diffusion model based on adversarial image information in a diffusion image generation service. Pattern (E) and below are any combination of patterns (A) to (D).

[0214] This allows the server 10 to set the destination of the hostile image use warning information within an appropriate range.

[0215] <Third Modification (4)> In the above embodiment, when a diffusion generated image is generated using an adversarial image, a diffusion generated image in which a watermark image is embedded is generated, but the present invention is not limited to this. For example, after a diffusion-generated image with an embedded watermark image is generated, if a user who generated the diffusion-generated image using a hostile image as a reference image in the diffusion image generation service obtains permission to use the original image from a user who registered the hostile image as a reference image in the diffusion image generation service or a user who generated the hostile image from an original image in the hostile image generation service, the reference image may be switched to the original image in the diffusion image generation service and the diffusion-generated image may be generated again.

[0216] In this case, in FIG. 3-3, for example, the hostile image registration information transmitted from the terminal 20A may include the hostile image and the original image. Then, the server 10 may store the hostile image and the original image in the reference image database. Of the two images, only the hostile image may be included in the reference image list information.

[0217] For example, when terminal 20B displays hostile image use warning information, terminal 20B may transmit original image use permission application information to server 10 based on a user input. Then, server 10 may transmit original image use permission request information to terminal 20A.

[0218] Terminal 20A may display the received original image usage permission request information and transmit the original image usage permission information based on a user input to server 10. Upon receiving the original image usage permission information, server 10 may execute the diffused generated image generation process again using the original image as a reference image, and transmit diffused generated image information in which a watermark image is not embedded to terminal 20B.

[0219] This allows the user of terminal 20B to generate a diffused generated image and confirm whether the generated image matches the user's intention before obtaining permission to use from the user of terminal 20A. After confirming that a diffused generated image that matches the user's intention has been generated, the user of terminal 20B can apply for permission to use from the user of terminal 20A and obtain a diffused generated image without a watermark.

[0220] Furthermore, the user of terminal 20A can spread the impression of a diffusion-generated image based on an adversarial image by publishing the adversarial image without issuing a license. And by releasing the original image as a reference image only to users who request a license, the process of accepting a license each time can be omitted. This allows users to widely and easily publish their own works while effectively preventing copyright infringement.

[0221] <Other>

[0222] At least a part of the processing that is to be performed by the server 10 in the above embodiment may be performed by the terminal 20. Conversely, at least a part of the processing that is to be performed by the terminal 20 in the above example may be performed by the server 10.

[0223] The operator of the server 10 may also be a provider of an adversarial image generation service or a viral image generation service in cooperation with a messaging service provider. In this case, the processing described in the above embodiments may be realized by one server, or the processing described in the above embodiments may be shared and realized by a server system consisting of multiple servers. A system configured with one or more servers may be defined as a server system, and the server of the present invention may be considered as a server system.

[0224] Furthermore, in the above-described embodiments, a server for distributing various applications (a server from which the terminal 20 downloads applications) may be configured as a server different from a server for providing the corresponding service (application). In other words, a server for distributing applications and a server for performing application management processing, etc., described in the above-described embodiments, etc., may be configured as physically separated servers, or may be configured as a single server.

[0225] Furthermore, applications are not limited to various application programs, but may also include, for example, a program that provides the functionality of another service as one function of a base application (for example, a program that provides the functionality of an adversarial image generation service or a viral image generation service as one function of a messaging application, or vice versa), a program for updating the base application, etc. Data used in application programs (which may include data for updating applications, etc.) may also be included.

[0226] In the above embodiments, the present invention is implemented by a client-server system, but is not limited to this. As mentioned above, the present invention may be implemented by a system such as a distributed system in which the terminal 20 has the functions of a server or server system. For example, the processes described in the flowcharts of the above embodiments as being performed by a server or server system may be performed by a terminal.

[0227] Furthermore, as mentioned above, the contents described in the above-mentioned embodiments, modifications, other embodiments, etc. can be applied in combination with each other. [Explanation of symbols]

[0228] 1. Information processing equipment 100 Communication Systems 10 Servers 20 terminals 30 Network

Claims

1. An information processing device capable of generating an adversarial image in a diffusion model, a control unit that generates the adversarial image based on a first image and a watermark image including identification information; the adversarial image is an image in which, when a second image is generated based on the diffusion model and the adversarial image, the identification information is visible in the second image.

2. 2. The information processing device according to claim 1, The information processing device, wherein the identification information includes information capable of identifying a user who has authority to use the first image as input to the diffusion model or authority to use the first image for additional learning of the diffusion model.

3. 3. The information processing device according to claim 2, The information processing device, wherein the identification information includes one or more of a character string, a signature, a code, and a fingerprint.

4. 2. The information processing device according to claim 1, The control unit a noise generating unit that generates noise based on the first image and the watermark image; An information processing device that generates the adversarial image based on the noise and the first image.

5. 5. The information processing device according to claim 4, The control unit calculating a first constraint relating to the first image and the adversarial image; calculating a second constraint relating to the watermark image and the second image; The noise generating unit is optimized based on the first constraint and the second constraint.

6. 6. The information processing device according to claim 5, The information processing device, wherein the noise generation unit is optimized based on weighting related to the first constraint condition and the second constraint condition.

7. 7. The information processing device according to claim 1, The adversarial image is an image in which, when the second image is generated based on inputting the adversarial image into the diffusion model, the identification information is visible in the second image.

8. 7. The information processing device according to claim 1, the adversarial image is an image in which the identification information is visible in the second image when the second image is generated based on the diffusion model in which the adversarial image was used for additional learning.

9. 7. The information processing device according to claim 1, the adversarial image is an image in which, when the second image is generated based on inputting the adversarial image into the diffusion model, the identification information is visible in the second image, and when a third image is generated based on the diffusion model in which the adversarial image is used for additional learning, the identification information is visible in the third image.

10. a server in communication with at least a first terminal, The control unit of the server includes the information processing device according to claim 1 , The control unit of the server receiving first information relating to the first image and the watermark image from the first terminal by a communication unit of the server; performing an association process based on a first account of the first terminal and the watermark image; generating the adversarial image; Second information regarding the hostile image is transmitted to the first terminal by the communication unit.

11. 11. The server of claim 10, The control unit of the server receiving the hostile image from a second terminal by a communication unit of the server; generating the second image; transmitting the second image to the second terminal by the communication unit; If the second image includes the watermark image associated with the first account, information about the first account is sent to the second terminal.

12. An information processing method in an information processing device capable of generating an adversarial image in a diffusion model, generating the adversarial image based on a first image and a watermark image including identification information; the adversarial image is an image in which, when a second image is generated based on the diffusion model and the adversarial image, the identification information is visible in the second image.

13. A program executed by an information processing device capable of generating an adversarial image in a diffusion model, generating the hostile image by a control unit of the information processing device based on a first image and a watermark image including identification information; The adversarial image is an image in which, when a second image is generated based on the diffusion model and the adversarial image, the identification information is visible in the second image.

Images