Information processing device, server, information processing method, and program

Abstract

The present invention improves convenience related to use of a diffusion model. This information processing device capable of generating an adversarial image in a diffusion model comprises a control unit that generates an adversarial image on the basis of a first image and a watermark image including identification information. The adversarial image is an image in which identification information can be visually recognized in a second image when the second image is generated on the basis of the diffusion model and the adversarial image.

Description

Information processing device, server, information processing method, and program

[0001] The present disclosure relates to an information processing device, a server, an information processing method, a program, and the like.

[0002] With the rapid development and widespread use of generative AI based on diffusion models (DMs), copyright infringement related to products generated by generative AI has become a concern. For example, Non-Patent Document 1 discloses a diffusion model capable of embedding an invisible digital watermark in an image. It also discloses a model that recovers a signature linked to an image by detecting the digital watermark from the embedded digital watermark. However, the method disclosed in Non-Patent Document 1 requires retraining the diffusion model, which requires time and computational resources for processing. Furthermore, a mechanical detection process for detecting the invisible digital watermark is essential to recover the signature.

[0003] Furthermore, for example, Non-Patent Document 2 discloses an algorithm for generating adversarial images to protect original images from learning and imitation using a diffusion model. However, while the method disclosed in Non-Patent Document 2 can prevent the generation of imitation images using a diffusion model, it cannot embed information about the author of the image.

[0004] Pierre Fernandez, et al. “The Stable Signature: Rooting Watermarks in Latent Diffusion Models”, 2023. Chumeng Liang, et al. “Adversarial Examples Does Good: Preventing Painting Imitation from Diffusion Models via Adversarial Examples”, 2023.

[0005] The present invention has been made in light of the above-mentioned technical background, and aims to provide an information processing method for generating an adversarial image in which a visible digital watermark is embedded in an image generated by a diffusion model, without retraining the diffusion model.

[0006] According to a first aspect of the present invention, an information processing device capable of generating an adversarial image in a diffusion model includes a control unit that generates the adversarial image based on a first image and a watermark image including identification information, wherein the adversarial image is an image in which the identification information is visible when a second image is generated based on the diffusion model and the adversarial image. According to a second aspect of the present invention, an information processing method in an information processing device capable of generating an adversarial image in a diffusion model includes generating an adversarial image based on a first image and a watermark image including identification information, wherein the adversarial image is an image in which the identification information is visible when the second image is generated based on the diffusion model and the adversarial image. According to a third aspect of the present invention, a program executed by an information processing device capable of generating an adversarial image in a diffusion model includes generating an adversarial image by a control unit of the information processing device based on the first image and a watermark image including identification information, wherein the adversarial image is an image in which the identification information is visible when the second image is generated based on the diffusion model and the adversarial image.

[0007] 1 is a diagram showing an example of the configuration of an information processing device according to a first embodiment. FIG. 2 is a flowchart showing an example of the flow of processing performed by the information processing device according to the first embodiment. FIG. 3 is a diagram showing an example of an image generated by a diffusion model from an adversarial image by the information processing device according to the first embodiment. FIG. 4 is a diagram showing an example of an image generated by a diffusion model that has learned adversarial images by the information processing device according to the first embodiment. FIG. 5 is a diagram showing an example of the system configuration of a communication system according to a second embodiment. FIG. 6 is a diagram showing an example of functions implemented by a control unit of a server according to the second embodiment. FIG. 7 is a diagram showing an example of information stored in a memory unit of a server according to the second embodiment. FIG. 8 is a diagram showing an example of account registration data according to the second embodiment. FIG. 9 is a diagram showing an example of functions implemented by a control unit of a terminal according to the second embodiment. FIG. 10 is a diagram showing an example of information stored in a memory unit of a terminal according to the second embodiment. FIG. 11 is a diagram showing an example of a screen displayed on a display unit of a terminal according to the second embodiment. FIG. 12 is a flowchart showing an example of the flow of processing performed by each device according to the second embodiment. FIG. 13 is a diagram showing an example of a screen displayed on a display unit of a terminal according to the third embodiment. FIG. 14 is a diagram showing an example of a screen displayed on a display unit of a terminal according to the third embodiment. FIG. 15 is a flowchart showing an example of the flow of processing performed by each device according to the third embodiment. A table showing an example of destinations of hostile image use warning information according to the third embodiment.

[0008] Compliance with Legal Matters It should be noted that the disclosure described herein is subject to compliance with the laws of the country of implementation necessary for the implementation of this disclosure, such as secrecy of communications.

[0009] <Embodiments> In this specification, for ease of understanding, there are places where the phrase "for example" is used, but please note that not only those places but also the entire embodiment described below are not limited to the contents of that description.

[0010] An embodiment for implementing a program etc. according to the present disclosure will be described with reference to the drawings.

[0011] The production of a terminal of the claimed invention (terminal of the claimed invention) may include, for example, the concept that a state is created in which the functions of the claimed invention can be realized (a state in which the claimed invention can be executed) on a terminal owned (possessed) by a user by receiving (or receiving and storing) a program (for example, an application program) described in this specification on the terminal.

[0012] Furthermore, the production of the system of the invention claimed in the present application (the system of the invention of the present application) may include, for example, the concept that a state in which the functions of the invention of the system claimed in the present application can be realized (a state in which the invention claimed in the present application can be executed) is created by receiving a program described in this specification (for example, an application program) transmitted from a server included in the system of the present application at a terminal included in the system of the present application (or by storing the received program in the terminal).

[0013] Furthermore, in this specification, a system can be, for example, a configuration including multiple devices. The multiple devices may be a combination of devices of the same type, a combination of devices of different types, or a combination of devices of the same type and devices of different types. Note that a system can also be, for example, considered to be a configuration in which multiple devices work together to perform some kind of processing.

[0014] Furthermore, a system relating to a client (client device) and a server can be considered to be, for example, at least one of the following: (1) Terminal and Server (2) Server (3) Terminal

[0015] (1) is, for example, a system including at least one terminal and at least one server. One example of this is a client-server system.

[0016] The server is configured by the following devices, for example, and may be a single device or a combination of multiple devices.

[0017] Specifically, the server is configured to have, for example, at least one processor (for example, CPU: Central Processing Unit, GPU: Graphics Processing Unit, APU: Accelerated Processing Unit, DSP: Digital Signal Processor (for example, ASIC: Application Specific Integrated Circuit, FPGA: Field Programmable Gate Array), etc.), computer device (processor + memory), control device, arithmetic device, processing device, etc., and may be configured to have multiple of the same type of device (for example, CPU + CPU, homogeneous multi-core processor, etc.), or multiple of different types of device (for example, CPU + DSP, heterogeneous multi-core processor, etc.), or may be a combination of multiple devices (for example, processor + computer device, processor + arithmetic device, multiple devices made heterogeneous, etc.). The processor may also be a virtual processor.

[0018] Furthermore, when a server performs some processing, if the server is configured with a single device, the processing described in the embodiments is performed by the single device. Furthermore, if the server is configured with multiple devices, some processing may be performed by one device and other processing may be performed by the other device. For example, if the server is configured with a processor and an arithmetic device, the processor may perform a first processing and the arithmetic device may perform a second processing. Furthermore, if the server is configured with multiple devices, the devices may be physically located apart from each other.

[0019] Furthermore, the functions of the server may be provided in the form of, for example, PaaS, IaaS, or SaaS in cloud computing. Some or all of the processes described in this specification may be implemented as a program included in an application installed on a terminal from the server. Furthermore, the manufacturer / manager of the terminal, the manufacturer / manager of the application, and the manufacturer / manager of the server may each be different entities (businesses), or some or all of them may be the same entity (business).

[0020] The control unit of the system can be at least one of the control unit of the terminal and the control unit of the server. That is, for example, the control unit of the system can be any of (1A) only the control unit of the terminal, (1B) only the control unit of the server, or (1C) both the control unit of the terminal and the control unit of the server.

[0021] Furthermore, the control and processing performed by the system's control unit (hereinafter collectively referred to as "control, etc.") may be (1A) performed only by the terminal's control unit, (1B) performed only by the server's control unit, or (1C) performed by both the terminal's control unit and the server's control unit. Also, in (1C), for example, some of the control, etc. performed by the system's control unit may be performed by the terminal's control unit, and the remaining control, etc. may be performed by the server's control unit. In this case, the allocation (allocation) of control, etc. may be equal, or may be allocated in different proportions.

[0022] Furthermore, when referring to the communication unit of a server, if the server is configured with a single device, it may refer to the communication unit itself that is included in the single device. Furthermore, if the server is configured with multiple devices, the communication unit of the server may be configured to include each communication unit that each device includes. For example, if the server includes a first device and a second device, and the first device has a first communication unit and the second device has a second communication unit, the communication unit of the server may be conceptually understood to include both the first communication unit and the second communication unit.

[0023] (2) can be, for example, a system consisting of multiple servers (hereinafter referred to as a "server system") In this case, the configuration of each server can be similarly applied to the configuration described above.

[0024] The control etc. performed by the server system may be performed by only one server (2A) among multiple servers, or by only other servers (2B), or by one server and other servers (2C). Also, in (2C), for example, some of the control etc. performed by the server system may be performed by one server, and the remaining control etc. may be performed by other servers. In this case, the allocation (allocation) of the control etc. may be equal, or may be allocated in different proportions.

[0025] (3) can be, for example, a system composed of multiple terminals. This system can be, for example, the following systems: - A system in which terminals have server functions (distributed system). This can be realized, for example, by using blockchain technology. - A system in which terminals communicate wirelessly with each other. This can be realized, for example, by communicating using a P2P (peer-to-peer) method using short-range wireless communication technology such as Bluetooth (registered trademark).

[0026] The above is not limited to the control unit, but also applies to each functional unit such as an input / output unit, a communication unit, a storage unit, and a clock unit that may be components of the system.

[0027] In the following embodiment, a system including a terminal and a server (a client-server system) is exemplified. Note that the server system described in (2) above can also be applied as the server.

[0028] Furthermore, instead of a system including a terminal and a server, it is also possible to apply a system that does not include a server, such as the system described in (3) above. In this case, the embodiment can be configured based on the blockchain technology described above. Specifically, for example, data stored and managed in the server described in the following embodiment is stored on the blockchain. Then, the terminal generates a transaction to the blockchain, and when the transaction is approved on the blockchain, the data stored on the blockchain is updated.

[0029] It should be noted that even when the term "terminal" is used, this does not mean that the terminal is limited to a client device in a client-server environment. In other words, the term "terminal" may also include the concept of a device that is not in a client-server environment.

[0030] Furthermore, in this specification, the expression "through a communication I / F" is used as appropriate. This may, for example, indicate that a device transmits and receives various information and data through a communication I / F (through a communication unit) based on the control of a control unit (such as a processor).

[0031] Furthermore, in this specification, when the terms "related to" or "related to" are used, for example, "B related to A" or "B related to A" may mean "B" that has some kind of relationship with "A." Specific examples of this will be described later.

[0032] Furthermore, in this specification, when a device processes two or more items, such as "transmitting A and B" or "receiving A and B," this may include both performing "A" and "B" in sync (hereinafter referred to as "simultaneous") and performing "A" and "B" at different times (hereinafter referred to as "non-simultaneous"). For example, when referring to transmitting first information and second information, this may include both transmitting the first information and the second information in sync and transmitting the first information and the second information at different times. Note that, taking into account lag (time lag), "simultaneous" may also include "almost simultaneously."

[0033] Note that, although "A" and "B" are performed at different times, this only needs to be performed for "A" and "B," and the purposes thereof do not necessarily have to be the same. For example, when transmitting first information and second information as described above, it is sufficient to transmit the first information and the second information, and this may include cases where the first information and the second information are transmitted for the same purpose, as well as cases where the first information and the second information are transmitted for different purposes.

[0034] An example of an embodiment of the present invention will be described below with reference to the drawings. In the description of the drawings, the same elements are designated by the same reference numerals, and duplicate descriptions may be omitted. Furthermore, the components described in this embodiment are merely examples, and are not intended to limit the scope of the present invention.

[0035] First Example The first example is an example in which an information processing device (which may also be called an image processing device, an image generation device, an adversarial image generation device, or an adversarial image generation unit) generates an adversarial image based on, for example, an original image and a watermark image. The content described in the first example is similarly applicable to any of the other examples and other modified examples.

[0036] 1-1 is a block diagram showing an example of a functional configuration of an information processing device 1 according to one aspect of this embodiment. The information processing device 1 includes, for example, a perturbation control unit 2, a diffusion model unit 3, an adversarial image generation unit 4, a differential loss calculation unit 5, and an adversarial loss calculation unit 6. These are, for example, functional units (functional blocks) included in a control unit (control device) (not shown) of the information processing device 1. The control unit may also be referred to as a processing unit (processing device).

[0037] The perturbation control unit 2 has a function of calculating a perturbation, which is a change that has a significant impact on an image (referred to as a "diffusion-generated image") generated in the diffusion model unit 3 while minimizing the impact on the original image "x" based on, for example, an original image 7 "x" and a watermark image 8 "m" input to the information processing device 1. The perturbation control unit 2 is configured, for example, with an encoder-decoder model such as an autoencoder or a U-Net. Hereinafter, the model weight of the perturbation control unit 2 (hereinafter, the "model weight" will be referred to as the "model weight") will be represented as "G", and the perturbation of the original image 7 "x" under the condition that the watermark image 8 "m" is given will be represented as "G(x|m)". The model weight may also be referred to as the connection weight between neurons in each model.

[0038] The diffusion model unit 3 is configured with latent diffusion models such as DDPM (Denoising Diffusion Probabilistic Models) and stable diffusion. In the diffusion model, for example, a reference image "y" is converted into a latent representation in a diffusion process. Then, the latent representation is converted into a diffusion-generated image in a de-diffusion process. Note that the diffusion model unit 3 may convert into a latent representation upon receiving a prompt. Hereinafter, the model weight of the diffusion model unit 3 is denoted as "θ", and the diffusion-generated image generated when the reference image "y" is given is denoted as "θ(y)".

[0039] The adversarial image generation unit 4 has a function of generating an adversarial image 9 "x'" based on, for example, a perturbation "G(x|m)" and an original image 7 "x." Here, the adversarial image 9 is an image that, when provided as an input to the diffusion model unit 3 as a reference image or an image for additional learning, has an unusual effect on the generation of a diffusion-generated image in the diffusion model unit 3 (interfering with the generation or adding noise or the like unintended by the generator).

[0040] The differential loss calculation unit 5 has a function of calculating, for example, a differential loss "Ldiff" to be used for adjusting the model weight of the perturbation control unit 2 based on the original image 7 "x", the adversarial image 9 "x'", and the watermark image 8 "m". In addition, the adversarial loss calculation unit 6 has a function of calculating, for example, an adversarial loss "Ladv" to be used for adjusting the model weight of the perturbation control unit 2 based on the diffusion generated image "θ(x')".

[0041] The original image 7 "x" is a tensor configured of the original image 7, which is, for example, a three-channel RGB image (color image) of "M x M" pixels ("M" is a natural number). The original image 7 "x" may also be a tensor configured of multiple original images 7 of "M x M" pixels. The original image 7 may also be a grayscale or black and white binary image (one-channel image).

[0042] The watermark image 8 "m" is a tensor formed by the watermark image 8, which is, for example, a three-channel image of "M x M" pixels. Note that the watermark image 8 "m" may also be a tensor formed by a one-channel image of "M x M" pixels. The watermark image 8 and each original image 7 may also be of different sizes.

[0043] The adversarial image 9 "x'" is, for example, a tensor composed of the adversarial image 9 composed of a three-channel RGB image of "M x M" pixels. Note that the adversarial image 9 "x'" may be a tensor composed of any number of adversarial images 9. The adversarial image 9 may also be a grayscale or black and white binary image. The original image 7 and the adversarial image 9 may also be different sizes.

[0044] The original image 7, the watermark image 8, and the adversarial image 9 may have rectangular sizes. Then, for example, before converting the image into a tensor, the image size may be normalized to a square.

[0045] 1-2 is a flowchart showing an example of the flow of processing executed by the information processing device 1 in this embodiment. Note that the processing described below is merely an example of processing for realizing the method of the present disclosure, and is not limited to this. Furthermore, other steps may be added to the processing described below, or some steps may be omitted (deleted) from the processing described below.

[0046] First, the control unit (not shown) of the information processing device 1 executes an original image acquisition process (P110). In the original image acquisition process, for example, the information processing device 1 acquires an arbitrary number of original images 7 and converts them into a tensor of the original images 7 "x".

[0047] Furthermore, the control unit of the information processing device 1 executes a watermark image acquisition process (P120). In the watermark image acquisition process, for example, when the information processing device 1 acquires the watermark image 8, it converts it into a tensor of the watermark image 8 "m".

[0048] Here, the following types of images may be used as the watermark image: A character string image (for example, the name of the author or copyright holder of the original image 7) A logo mark image or signature image (for example, the signature of the author or copyright holder of the original image 7) A fingerprint image (for example, the fingerprint of the author or copyright holder of the original image 7) A code image (for example, a barcode indicating an ID uniquely associated with the author or copyright holder of the original image 7) This information may be said to be identification information that is included in the watermark image and is visible to the user.

[0049] The identification information included in the watermark image may be said to be information for identifying a user who has the authority to generate an image (based on the original image) that does not include a watermark image as a diffusion-generated image of the diffusion model unit 3. Here, the generation of an image based on the original image may be (i) generating a diffusion-generated image based on the original image 7 without additional training of the diffusion model unit 3, i.e., generating a diffusion-generated image by inputting the original image 7 to the diffusion model unit 3 as a reference image. Alternatively, (ii) generating a diffusion-generated image based on additional training of the diffusion model unit 3, i.e., generating a diffusion-generated image by using the original image 7 for additional training of the diffusion model unit 3 and inputting a reference image to the diffusion model unit 3 after the additional training. Alternatively, both (i) and (ii) may be included. In other words, the identification information included in the watermark image may be identification information that can identify a user who has the authority to use the original image 7 as input to the diffusion model unit 3 or the authority to use it for additional training of the diffusion model unit 3.

[0050] If the original image 7 and the watermark image 8 are different in size, the information processing device 1 may, for example, adjust (enlarge or reduce) the size of the original image 7 or the watermark image 8 to match the sizes of the original image 7 and the watermark image 8, and then convert them into tensors. Also, if the original image 7 and the watermark image 8 have different numbers of channels, the information processing device 1 may, for example, convert the images into images with fewer channels (for example, converting a color image into a grayscale image), match the numbers of channels, and then convert them into tensors.

[0051] Here, "acquisition" of an image can include not only acquisition (input) of an image on the device itself, but also, for example, input of an image from another functional unit (e.g., a memory unit not shown) on the device itself (internal input), input of an image (external input) or reception (external reception) from a device other than the device itself (external device), etc.

[0052] Then, the control unit of the information processing device 1 executes a perturbation calculation process (P130). In the perturbation calculation process, for example, the perturbation control unit 2 connects "x" and "m" for each channel and accepts them as input. This allows the perturbation, which is the output of the perturbation control unit 2, to be calculated using "m" as a condition for the input "x". Then, the perturbation control unit 2 calculates a perturbation "G(x|m)" according to the model weight "G".

[0053] Then, the control unit of the information processing device 1 executes an adversarial image generation process (P140). In the adversarial image generation process, for example, the adversarial image generation unit 4 adds the original image 7 "x" and the perturbation "G(x|m)" to generate an adversarial image 9 "x'" = "x" + "G(x|m)".

[0054] Here, the perturbation "G(x|m)" is a change that has a large effect on the diffusion generated image "θ(x')" of the diffusion model unit 3 without having a large apparent effect on the original image 7 "x".

[0055] Therefore, the control unit of the information processing device 1 executes a differential loss calculation process (P150). In the differential loss calculation process, for example, the differential loss calculation unit 5 calculates the differential loss according to the following formula. Here, "Ex(x)" denotes the expected value of "x" (e.g., arithmetic mean). Also, "||x||" denotes the norm of "x" (e.g., L2 norm). In other words, the differential loss takes a smaller value the more similar the original image 7 "x" and the adversarial image 9 "x'" are. It can be said that the differential loss is a loss term that prevents the adversarial image 9 "x'" from becoming an image that is too different from the original image 7 "x".

[0056] When the hostile image is generated, the control unit of the information processing device 1 executes a diffusion image generation process based on the hostile image (P160). In the diffusion image generation process, for example, the diffusion model unit 3 inputs the hostile image 9 “x′” as a reference image and generates a diffusion generated image “θ(x′).”

[0057] Then, the control unit of the information processing device 1 executes the adversarial loss calculation process (P170). In the adversarial loss calculation process, for example, the adversarial loss calculation unit 6 calculates the differential loss according to the following formula. That is, the more similar the diffusion-generated image "θ(x')" is to the watermark image 8 "m", the smaller the value of the adversarial loss. It can be said that the adversarial loss is a loss term for making the diffusion-generated image "θ(x')" based on the adversarial image 9 "x'" approach the watermark image 8 "m".

[0058] Then, the control unit of the information processing device 1 executes an optimization loss calculation process (P180). The optimization loss "L" is, for example, a loss used to optimize the model weight "G" in the perturbation control unit 2, and may be calculated according to the following formula:

[0059] Here, “α” and “β” are parameters for balancing the adversarial loss and the differential loss. For example, “α” and “β” may be values ​​greater than “0.” Alternatively, “α + β = 1” may be used. Alternatively, “α = 1” and “β = 10,” or “α = 10” and “β = 1” may be used. When “α” is large, the adversarial loss term dominates the optimization loss. Therefore, by setting “α” larger than “β,” the success rate of attacks in the diffusion model unit 3 can be improved. Conversely, when “β” is large, the differential loss term dominates the optimization loss. Therefore, by setting “β” larger than “α,” the noise in the adversarial image 9 “x′” can be reduced (e.g., the S / N ratio can be improved). By adjusting the parameters “α” and “β,” it is possible to achieve a balance between the magnitude of the noise in the adversarial image 9 “x′” and the effectiveness of attacks against the diffusion model unit 3.

[0060] The parameters “α” and “β” can be said to be parameters for adjusting the visibility of the identification information included in the watermark image 8 in the adversarial image 9 .

[0061] When the optimization loss is calculated, the control unit of the information processing device 1 determines, for example, whether the optimization loss is equal to or greater than a preset threshold value (P190). Note that the control unit of the information processing device 1 may also determine, for example, whether the optimization loss is greater than a preset threshold value.

[0062] If it is determined that the optimization loss is equal to or greater than the threshold (P190: YES), the control unit of the information processing device 1 executes a perturbation control unit model weight adjustment process (P200). In the perturbation control unit model weight adjustment process, for example, the perturbation control unit 2 adjusts the model weight "G" by backpropagation so as to minimize the optimization loss "L". Note that the perturbation control unit 2 may also adjust the model weight "G" by an optimization method such as a genetic algorithm (GA).

[0063] After adjusting the model weight "G", for example, the control unit of the information processing device 1 returns the process to P130.

[0064] When it is determined that the optimization loss is smaller than the threshold value (P190: NO), the control unit of the information processing device 1 executes an adversarial image output process (P210).

[0065] The control unit of the information processing device 1 may, for example, shift the process to the adversarial image output process (P210) when the number of executions of the optimization loss calculation process is equal to or exceeds a predetermined number. In this case, the control unit of the information processing device 1 may output an optimization failure notification indicating that the perturbation control unit 2 has failed to optimize the model weight "G" and interrupt the process. Alternatively, the control unit may change and reset the parameters "α" and "β" and re-execute the process from P180.

[0066] In the adversarial image output process, for example, the control unit of the information processing device 1 converts the adversarial image 9 “x′” generated in the adversarial image generation process from a tensor to an image, and outputs it as an adversarial image for the diffusion model unit 3. Then, the control unit of the information processing device 1 ends the process.

[0067] In the adversarial image output process, for example, the control unit of the information processing device 1 may output the optimized model weight “G” of the perturbation control unit 2 .

[0068] Here, "output" of an image can include not only displaying the image on the device itself (display output), but also, for example, outputting the image to another functional unit on the device itself (internal output), outputting the image to a device other than the device itself (external device) (external output) or transmitting the image (external transmission), etc.

[0069] The processing in P130 to P200 is processing for optimizing the model weight "G" of the perturbation control unit 2. Therefore, the processing in P130 to P200 may be collectively referred to as perturbation control unit optimization processing or perturbation control unit learning processing.

[0070] Furthermore, if the model weight “G” of the perturbation control unit 2 optimized for the original image 7 and the watermark image 8 has already been calculated, the control unit of the information processing device 1 may output the adversarial image 9 without performing the perturbation control unit optimization process. In this case, for example, the control unit of the information processing device 1 executes the original image acquisition process and the watermark image acquisition process to obtain the optimized model weight “G.” Then, the control unit of the information processing device 1 executes the perturbation calculation process using the optimized model weight “G.” This process may also be referred to as a perturbation control unit inference process. Then, the control unit of the information processing device 1 may execute the adversarial image generation process based on the generated perturbation, and output the generated adversarial image 9 in the adversarial image output process. In this way, by executing the perturbation control unit inference process without performing the perturbation control unit optimization process, the adversarial image 9 based on the original image 7 and the watermark image 8 can be quickly generated.

[0071] <Example of Generation of Diffusionally Generated Images Based on Adversarial Images> When an adversarial image in this embodiment is used in the diffusion model, for example, the methods of using the adversarial image can be broadly categorized as follows: Diffusionally generated image generation method (A): When an adversarial image is used as a reference image. The diffusion model unit 3 generates a diffusely generated image based on, for example, an adversarial image and a prompt. Diffusionally generated image generation method (B): When an adversarial image is used as an image for additional learning (fine tuning). The model weight "θ" of the diffusion model unit 3 is perturbed to a model weight "θ'" by additional learning using an adversarial image, but the diffusely generated image "θ'(x')" is still an image close to the original image 7 "x".

[0072] Figure 1-3 shows three examples (A) to (C) of diffuse generated images generated based on adversarial images using the diffuse generated image generation method (A). In Figure 1-3, the watermark image 8 uses the character string images "IMAGENET_CAT," "IMAGENET_DOG," and "IMAGENET_SHARK," respectively, from left to right. The perturbation control unit optimization process using 5 to 10 samples as the original image required 2 to 3 minutes of processing time when executed on an NVIDIA A100 80GB GPU. After the perturbation control unit optimization process, the adversarial image generation process required 0.2 seconds to generate one adversarial image.

[0073] 1-3 show, from top to bottom, an original image 7, an adversarial image 9 generated using the original image 7 and a watermark image 8, a diffusion-generated image generated in the diffusion model unit 3 using the original image 7 as a reference image, and a diffusion-generated image generated in the diffusion model unit 3 using the adversarial image as a reference image. The prompt for generating the diffusion-generated image was "A painting." Each image was an RGB image of 512 x 512 pixels.

[0074] From these examples, it can be seen that the original image 7 and the adversarial image 9 are so similar that they are virtually indistinguishable visually. Furthermore, it can be seen that the character string image of the watermark image 8 is generated in a visibly emerging state in the diffusion generated image generated using the adversarial image 9 as the reference image. In the adversarial image 9, the visibility of the identification information contained in the watermark image 8 is low (or not visible), but in the diffusion generated image generated using the adversarial image 9 as the reference image, the visibility of the identification information contained in the watermark image 8 is significantly improved compared to the adversarial image 9. In other words, a user can detect the identification information contained in the watermark image 8 by visually checking the diffusion generated image.

[0075] FIG. 1-4 shows an example of a generation result when the diffusion model unit 3 is fine-tuned using an adversarial image 9 using, for example, the Textual Inversion method in the Stable Diffusion model in the diffusion-generated image generation method (B). In Textual Inversion, for example, when an image for additional learning is given, the diffusion model unit 3 learns to express the concept of the given image with the word "S*." Then, a diffusion-generated image is generated using "S*" representing the acquired concept as a prompt.

[0076] The left side of Figure 1-4 shows a diffusion generated image generated by using the prompt "An oil painting of S*" and the prompt "A photo of S* on a boat" in the diffusion model unit 3 after additional learning when the original image 7 is used as an image for additional learning. When the original image 7 is additionally learned by textual inversion, a diffusion generated image is generated in which the original image 7 is the concept "S*".

[0077] The right side of Figure 1-4 shows a diffusion-generated image generated by the same prompt in the diffusion model unit 3 after additional training when an adversarial image 9 is used as an image for additional training. The watermark image 8 used to generate the adversarial image 9 is a character string image of "IMAGENET_CAT." It can be seen that the character string image of the watermark image 8 is generated in a visibly visible manner in the diffusion-generated image generated using the diffusion model that has been additionally trained with the adversarial image 9. In other words, it can be seen that the adversarial image 9 generated by this method is effective not only when used as a reference image, but also when used as an image for additional training.

[0078] <Effects of First Example> According to this example, by generating an adversarial image 9 based on an original image 7 and a watermark image 8 including identification information, when a diffusion-generated image is generated by inputting the adversarial image 9 as a reference image in a diffusion model, the identification information included in the watermark image 8 can be visually recognized in the diffusion-generated image. Furthermore, according to this example, when the adversarial image 9 is used for additional learning of the diffusion model, and a diffusion-generated image is generated by inputting a reference image (the adversarial image 9 or an image different from the adversarial image 9) into the diffusion model after the additional learning has been performed, the identification information included in the watermark image 8 can be visually recognized in the diffusion-generated image.

[0079] That is, it is possible to easily identify from the diffusion-generated image whether the diffusion-generated image was generated by a "user who does not have the authority to use the original image 7 as a reference image for the diffusion model" by inputting the adversarial image 9 into the diffusion model as a reference image for the diffusion model, or whether the diffusion-generated image was generated by a "user who does not have the authority to use the original image 7 for additional training of the diffusion model" by using the adversarial image 9 for additional training of the diffusion model and inputting the reference image into the diffusion model after the additional training. This makes it possible to prevent the adversarial image 9 from being used for purposes not intended by the author of the original image 7, etc.

[0080] Second Example In the second example, an adversarial image 9 is generated in a server 10 equipped with the information processing device 1 described in the first example, using an original image 7 and a watermark image 8 provided (specified) by a user of a terminal 20.

[0081] The contents described in the second embodiment can be similarly applied to any of the other embodiments and other modified examples.

[0082] In the following embodiments, a service for realizing the generation of the adversarial image 9 will be referred to as an "adversarial image generation service" as an example. The adversarial image generation service may be usable together with an image generation service using a diffusion model (referred to as a "diffuse image generation service"). An application for realizing the adversarial image generation service will be referred to as an "adversarial image generation application."

[0083] In the following description, the user of terminal 20A communicating with server 10 will be referred to as "user A.A.", the user of terminal 20B as "user B.B.", the user of terminal 20C as "user C.C.", and so on.

[0084] <System Configuration> Figure 2-1 is a diagram showing an example of the system configuration of the communication system 100 in this embodiment. In the communication system 100, for example, a server 10 is connected to one or more terminals 20 (terminal 20A, terminal 20B, terminal 20C, ...) via a network 30.

[0085] The server 10 has a function of providing, for example, an adversarial image generation service to a terminal 20 owned by a user via a network 30. The server 10 can also be expressed as an adversarial image generation service server or the like. In this embodiment, for example, the user of the server 10 is a company that provides an adversarial image generation service (an operator of the adversarial image generation service). Note that the number of servers 10 and the number of terminals 20 connected to the network 30 are not limited to the above. In this embodiment, the "adversarial image generation service" is a service provided by an operator (server 10) such as a company that provides an adversarial image generation service, and may be provided to a user (user's terminal 20), for example.

[0086] The terminal 20 (terminal 20A, terminal 20B, terminal 20C, etc.) may be any information processing terminal capable of implementing the functions described in each embodiment. Examples of the terminal 20 include smartphones, mobile phones (feature phones), computers (including, but not limited to, desktops, laptops, tablets, etc.), media computer platforms (including, but not limited to, cable and satellite set-top boxes, digital video recorders, etc.), handheld computer devices (including, but not limited to, personal digital assistants (PDAs), email clients, etc.), wearable devices (glasses-type devices, watch-type devices, etc.), virtual reality (VR) terminals, smart speakers (voice recognition devices), or other types of computers or communication platforms. The terminal 20 may also be referred to as an information processing terminal.

[0087] For example, the configurations of terminals 20A, 20B, and 20C may be the same. Furthermore, if necessary, the terminal used by user X may be referred to as terminal 20X, and user information in a predetermined service associated with user X or terminal 20X may or may not be referred to as user information X. Note that user information is user information associated with an account used by the user in a predetermined service. User information includes, by way of example and without limitation, information associated with a user, such as the user's name, user icon image, user age, user gender, user address, user hobbies and interests, and user identifier, which is input by the user or assigned by the predetermined service, and may be any one or a combination of these.

[0088] The network 30 serves to connect the devices constituting the communication system 100. That is, the network 30 refers to a communication network that provides connection paths so that the above-mentioned various devices can be connected and transmit and receive data.

[0089] One or more portions of network 30 may or may not be a wired or wireless network. Network 30 may include, by way of example, an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), a portion of the Internet, a portion of the Public Switched Telephone Network (PSTN), a cellular network, integrated service digital networks (ISDN), wireless LAN, long term evolution (LTE), code division multiple access (CDMA), Bluetooth, satellite communications, etc., or a combination of two or more thereof. Network 30 may include one or more networks 30.

[0090] The server 10 (which is not limited to a server, an example of an information processing device, or an information management device) has a function of providing a predetermined service (in this embodiment, an adversarial image generation service) to the terminal 20, etc. The server 10 may be any information processing device capable of implementing the functions described in each embodiment. Examples of the server 10 include a server device, a computer (e.g., a desktop, a laptop, a tablet, etc.), a media computer platform (e.g., a cable or satellite set-top box, a digital video recorder, etc.), a handheld computer device (e.g., a PDA, an email client, etc.), or other types of computers or communication platforms. The server 10 may also be referred to as an information processing device. When there is no need to distinguish between the server 10 and the terminal 20, the server 10 and the terminal 20 may or may not each be referred to as an information processing device.

[0091] [Hardware (HW) Configuration of Each Device] The HW configuration of each device included in the communication system 100 will be described.

[0092] (1) HW Configuration of Terminal FIG. 2-1 shows an example of the HW configuration of the terminal 20. The terminal 20 includes, for example, a control unit 21 (CPU: central processing unit), a storage unit 28, a communication I / F 22 (interface), an input / output unit 23, a clock unit 29A, and a position calculation information detection unit 29B. The HW components of the terminal 20 are connected to each other, for example, via a bus B. Note that it is not essential for the HW configuration of the terminal 20 to include all components. For example, the terminal 20 may or may not be configured so that individual components or multiple components can be removed.

[0093] The communication I / F 22 transmits and receives various data via the network 30. The communication may be performed either wired or wirelessly, and any communication protocol may be used as long as mutual communication is possible. The communication I / F 22 has a function of communicating with various devices, such as the server 10, via the network 30. The communication I / F 22 transmits various data to various devices, such as the server 10, in accordance with instructions from the control unit 21. The communication I / F 22 also receives various data transmitted from various devices, such as the server 10, and transmits it to the control unit 21. The communication I / F 22 may also be simply referred to as a communication unit. When the communication I / F 22 is configured as a physically structured circuit, it may also be referred to as a communication circuit.

[0094] The input / output unit 23 includes a device for inputting various operations to the terminal 20, a device for outputting processing results processed by the terminal 20, etc. The input / output unit 23 may be an integrated unit having an input unit and an output unit, or may be separate units having an input unit and an output unit, or may not be so.

[0095] The input unit is realized by any one or a combination of all types of devices that can accept input from a user and transmit information related to the input to the control unit 21. Examples of the input unit include hardware keys such as a touch panel, a touch display, and a keyboard, a pointing device such as a mouse, a camera (for inputting operations via moving images), and a microphone (for inputting operations via voice).

[0096] The output unit is realized by any one or a combination of all types of devices that can output the processing results processed by the control unit 21. Examples of the output unit include a touch panel, a touch display, a speaker (audio output), a lens (for example, 3D (three dimensions) output or hologram output), a printer, etc.

[0097] Although this is merely an example, the input / output unit 23 includes a display unit 24 , a sound input unit 25 , a sound output unit 26 , and an imaging unit 27 .

[0098] The display unit 24 is realized by any one of all types of devices or a combination thereof that can display according to the display data written to the frame buffer. Examples of the display unit 24 include a touch panel, a touch display, a monitor (e.g., a liquid crystal display or an organic electroluminescence display (OELD)), a head mounted display (HDM), a projection mapping, a hologram, and a device that can display images, text information, etc. in air (which may or may not be a vacuum). Note that these display units 24 may or may not be capable of displaying display data in 3D.

[0099] The sound input unit 25 is used to input sound data (including voice data; the same applies below). The sound input unit 25 includes a microphone and the like. The sound output unit 26 is used to output sound data. The sound output unit 26 includes a speaker and the like. The imaging unit 27 is used to acquire image data (including still image data and moving image data; the same applies below). The imaging unit 27 includes a camera and the like.

[0100] When the input / output unit 23 is a touch panel, the input / output unit 23 and the display unit 24 may be disposed opposite each other and have approximately the same size and shape.

[0101] The clock unit 29A is a built-in clock of the terminal 20 and outputs time information (timekeeping information). The clock unit 29A is configured, for example, with a clock that uses a crystal oscillator. The clock unit 29A can also be expressed, for example, as a timekeeping unit or a time information detection unit.

[0102] The clock unit 29A may or may not have a clock that conforms to the NITZ (Network Identity and Time Zone) standard or the like.

[0103] The position calculation information detection unit 29B is a functional unit that detects (measures) information (hereinafter referred to as "position calculation information") necessary for the control unit 21 to calculate (measure) the position of the own terminal 20. The position calculation information detection unit 29B can also be expressed as a position calculation sensor unit, for example.

[0104] The position calculation information detection unit 29B includes, for example, a satellite positioning sensor (satellite positioning unit) which is a sensor or unit for calculating the position of the terminal 20 using a satellite positioning system such as GPS (Global Positioning System), an inertial measurement sensor (inertial measurement unit (IMU (Inertial Measurement Unit))) which is a sensor or unit for calculating the position of the terminal 20 using an inertial navigation system, a UWB positioning sensor (UWB positioning unit) which is a sensor or unit for calculating the position of the terminal 20 using UWB (Ultra Wide Band), and the like.

[0105] The satellite positioning unit includes, for example, an RF receiving circuit that converts RF (Radio Frequency) signals, including positioning satellite signals transmitted from positioning satellites and received by an antenna (not shown), into digital signals, and a baseband processing circuit that performs correlation calculation processing on the digital signals output from the RF receiving circuit to capture the positioning satellite signals, and outputs information such as satellite orbit data and time data extracted from the positioning satellite signals as information for position calculation.

[0106] The inertial measurement unit has an inertial sensor that is a sensor that detects information necessary for calculating the position of the terminal 20 by inertial navigation calculation. The inertial sensor includes, for example, a three-axis acceleration sensor and a three-axis gyro sensor, and outputs the acceleration detected by the acceleration sensor and the angular velocity detected by the gyro sensor as information for position calculation.

[0107] The UWB positioning unit includes, for example, an ultra-wideband RF (Radio Frequency) receiving circuit that converts an ultra-wideband RF signal, including an ultra-wideband pulse signal for positioning transmitted from a positioning beacon and received by an antenna (not shown), into a digital signal, and a relative position calculation processing circuit that calculates the relative position between the terminal 20 and the positioning beacon based on the digital signal output from the ultra-wideband RF receiving circuit. Note that, for example, the UWB positioning unit may or may not cause the terminal 20 to function as a positioning beacon by transmitting an ultra-wideband RF signal including an ultra-wideband pulse signal for positioning from an antenna (not shown).

[0108] For example, the control unit 21 calculates the position of its own terminal 20 at regular intervals or specific intervals based on the position calculation information detected by the position calculation information detection unit 29B. The position of the terminal is referred to as the "terminal position," and the calculated terminal position is referred to as the "calculated terminal position." The control unit 21 may, but need not, associate the calculated terminal position with the date and time when the calculated terminal position is calculated and store the calculated terminal position in the storage unit 28 as calculated terminal position history data.

[0109] The control unit 21 has a circuit physically structured to execute the functions realized by the code or instructions contained in the program, and is realized by, for example, a data processing device built into hardware. Therefore, the control unit 21 may or may not be expressed as a control circuit.

[0110] The control unit 21 includes, for example, a central processing unit (CPU), a microprocessor, a processor core, a multiprocessor, an application-specific integrated circuit (ASIC), or a field programmable gate array (FPGA).

[0111] The storage unit 28 has a function of storing various programs and various data required for the operation of the terminal 20. The storage unit 28 includes, for example, various storage media such as a hard disk drive (HDD), a solid state drive (SSD), a flash memory, a random access memory (RAM), and a read only memory (ROM). The storage unit 28 may or may not be expressed as a memory.

[0112] Terminal 20 stores program P in storage unit 28, and by executing this program P, control unit 21 executes the processes of each unit included in control unit 21. In other words, program P stored in storage unit 28 causes terminal 20 to realize each function executed by control unit 21. Furthermore, this program P may or may not be expressed as a program module.

[0113] (2) Server HW Configuration FIG. 2-1 shows an example of the HW configuration of the server 10. The server 10 includes, for example, a control unit 11 (CPU), a memory unit 15, a communication I / F 14 (interface), an input / output unit 12, a display unit 13, and a clock unit 19. The HW components of the server 10 are connected to each other, for example, via a bus B. Note that the HW of the server 10 does not necessarily have to include all of the components. For example, the HW of the server 10 may or may not be configured so that individual components or multiple components can be removed.

[0114] The control unit 11 has circuits that are physically structured to execute the functions realized by the codes or instructions contained in the program, and is realized, for example, by a data processing device built into hardware.

[0115] The control unit 11 is typically a central processing unit (CPU), but may also be a microprocessor, a processor core, a multiprocessor, an ASIC, an FPGA, or other devices. In the present disclosure, the control unit 11 is not limited to these.

[0116] The storage unit 15 has a function of storing various programs and various data required for the operation of the server 10. The storage unit 15 is realized by various storage media such as an HDD, an SSD, or a flash memory. However, in the present disclosure, the storage unit 15 is not limited to these. Furthermore, the storage unit 15 may or may not be expressed as a memory.

[0117] The communication I / F 14 transmits and receives various data via the network 30. The communication may be performed either wired or wirelessly, and any communication protocol may be used as long as mutual communication is possible. The communication I / F 14 has a function of communicating with various devices, such as the terminal 20, via the network 30. The communication I / F 14 transmits various data to various devices, such as the terminal 20, in accordance with instructions from the control unit 11. The communication I / F 14 also receives various data transmitted from various devices, such as the terminal 20, and transmits it to the control unit 11. The communication I / F 14 may also be simply referred to as a communication unit. When the communication I / F 14 is configured as a physically structured circuit, it may also be referred to as a communication circuit.

[0118] The input / output unit 12 includes a device for inputting various operations to the server 10, a device for outputting processing results processed by the server 10, etc. The input / output unit 12 may be an integrated input unit and an output unit, or may be separate input unit and output unit, or may not be the same.

[0119] The input unit is realized by any one of or a combination of all types of devices that can accept input from a user and transmit information related to the input to the control unit 11. The input unit is typically realized by hardware keys such as a keyboard or a pointing device such as a mouse. Note that the input unit may or may not include, for example, a touch panel, a camera (for operation input via moving images), or a microphone (for operation input via voice).

[0120] The output unit is realized by any one or a combination of all types of devices that can output the processing results processed by the control unit 11. Examples of the output unit include a touch panel, a touch display, a speaker (sound output), a lens (for example, 3D (three dimensions) output or hologram output), a printer, etc.

[0121] By way of example only, the input / output unit 12 includes a display unit 13, for example.

[0122] The display unit 13 is realized by a display or the like. The display is typically realized by a monitor (for example, a liquid crystal display or an organic electroluminescence display (OELD)). The display may or may not be a head-mounted display (HDM) or the like. These displays may or may not be capable of displaying display data in 3D. In the present disclosure, the display is not limited to these.

[0123] The clock unit 19 is a built-in clock of the server 10 and outputs time information (timekeeping information). The clock unit 19 is configured to include, for example, an RTC (Real Time Clock) as a hardware clock, a system clock, etc. The clock unit 19 can also be expressed as, for example, a timekeeping unit or a time information detection unit.

[0124] (3) Others The server 10 stores a program P in the storage unit 15, and by executing this program P, the control unit 11 executes the processes of each unit included in the control unit 11. In other words, the program P stored in the storage unit 15 causes the server 10 to realize each function executed by the control unit 11. This program P may or may not be expressed as a program module. The same applies to other devices.

[0125] In each embodiment of the present disclosure, the description will be given assuming that the CPU of the terminal 20 and / or the server 10 executes the program P to realize the present invention. The same applies to other devices.

[0126] The control unit 21 of the terminal 20 and / or the control unit 11 of the server 10 may or may not realize each process not only by a CPU having a control circuit, but also by a logic circuit (hardware) formed in an integrated circuit (IC (Integrated Circuit) chip, LSI (Large Scale Integration)), or a dedicated circuit. These circuits may be realized by one or more integrated circuits, and multiple processes shown in each embodiment may or may not be realized by a single integrated circuit. LSIs may also be referred to as VLSIs, super LSIs, ultra LSIs, etc., depending on the level of integration. Therefore, the control unit 21 may or may not be referred to as a control circuit. The same applies to other devices.

[0127] Furthermore, the program P (e.g., a software program, a computer program, or a program module) of each embodiment of the present disclosure may or may not be provided in a state stored in a computer-readable storage medium. The storage medium can store the program P in a "non-transitory tangible medium." The program P may or may not be intended to realize part of the functions of each embodiment of the present disclosure. Furthermore, the program P may or may not be a so-called difference file (difference program) that can realize the functions of each embodiment of the present disclosure in combination with a program P already recorded on a storage medium.

[0128] The storage medium may include one or more semiconductor-based or other integrated circuits (ICs) (such as, for example, field programmable gate arrays (FPGAs) or application-specific ICs (ASICs)), hard disk drives (HDDs), hybrid hard drives (HHDs), optical disks, optical disk drives (ODDs), magneto-optical disks, magneto-optical drives, floppy diskettes, floppy disk drives (FDDs), magnetic tapes, solid-state drives (SSDs), RAM drives, secure digital cards, or drives, any other suitable storage media, or any suitable combination of two or more thereof. The storage medium may be volatile, non-volatile, or a combination of volatile and non-volatile, where appropriate. Note that the storage medium is not limited to these examples and may be any device or medium capable of storing the program P. Furthermore, the storage medium may or may not be referred to as a memory.

[0129] The server 10 and / or the terminal 20 can realize the functions of the multiple functional units shown in each embodiment by reading the program P stored in the storage medium and executing the read program P. The same applies to other devices.

[0130] Furthermore, the program P of the present disclosure may or may not be provided to the server 10 and / or the terminal 20 via any transmission medium capable of transmitting a program (such as a communication network or broadcast waves). The server 10 and / or the terminal 20, for example, executes the program P downloaded via the Internet or the like to realize the functions of the multiple functional units shown in each embodiment. The same applies to other devices.

[0131] Furthermore, each embodiment of the present disclosure may be realized in the form of a data signal in which the program P is embodied by electronic transmission. At least a portion of the processing in the server 10 and / or the terminal 20 may, or may not, be realized by cloud computing consisting of one or more computers. At least a portion or all of the processing in the terminal 20 may, or may not, be configured to be performed by the server 10. In this case, at least a portion or all of the processing of each functional unit of the control unit 21 of the terminal 20 may, or may not, be configured to be performed by the server 10. At least a portion or all of the processing in the server 10 may, or may not, be configured to be performed by the terminal 20. In this case, at least a portion or all of the processing of each functional unit of the control unit 11 of the server 10 may, or may not, be configured to be performed by the terminal 20. Unless explicitly stated otherwise, a determination configuration in the embodiments of the present disclosure is not essential, and a predetermined process may, or may not, be executed when a determination condition is met, or a predetermined process may, or may not, be executed when a determination condition is not met.

[0132] The program of the present disclosure is implemented using, for example, a scripting language such as ActionScript or JavaScript (registered trademark), a compiler language such as Objective-C or Java (registered trademark), or a markup language such as HTML Living Standard.

[0133] [Functional Configuration of Each Device] (1) Functional Configuration of Server Fig. 2-2 is a diagram showing an example of functions realized by the control unit 11 of the server 10 in this embodiment. The control unit 11 includes, for example, an adversarial image generation unit 1 and an application management processing unit 111 as functional units.

[0134] The adversarial image generator 1 includes, for example, the functional units shown in FIG. 1A. The application management processor 111 has, for example, a function of executing application management processing in accordance with an application management processing program 151 stored in the storage unit 15.

[0135] 2-3 is a diagram illustrating an example of information stored in the storage unit 15 of the server 10 in this embodiment. The storage unit 15 stores, for example, an application management processing program 151 executed as application management processing, and account registration data 153.

[0136] The account registration data 153 is registration data related to the account of the application (in this embodiment, the adversarial image generation application), and an example of the data configuration is shown in Fig. 2-4. For example, the account registration data 153 stores a user name, an application ID, watermark image information, and other registration information in association with each other.

[0137] The user name is the name of the account of the terminal 20 that uses this application, and for example, the name that the user of the terminal 20 registers when using the application is stored.

[0138] The application ID is information used to identify an application account, or the account itself. This application ID is preferably a value unique to each account, and for example, a unique value (proper value) is set and stored for each account by the server 10. The application ID is information associated with the terminal 20 or the user of the terminal 20, and is an example of information related to the terminal or the user of the terminal.

[0139] The watermark image information is a watermark image 8 associated with a user. The watermark image information can be specified by the user and stored in the server 10. The watermark image information can also be called image author identification information or copyright information. The watermark image information can also be called image owner identification information or ownership information.

[0140] Note that a plurality of watermark images 8 may be associated with one application ID and stored as watermark image information. That is, a watermark image 8 may be prepared for each original image 7 (original image set), and a plurality of watermark images 8 for the plurality of original images 7 may be used to generate respective adversarial images 9.

[0141] Other registration information may include, for example, various types of information such as identification information for identifying terminal 20, the telephone number (terminal telephone number) of terminal 20, an email address (terminal email address), and authentication information such as passwords (login password, authentication password, etc.) used for various authentications in the application.

[0142] The identification information for identifying the terminal 20 may be, for example, a terminal ID (for example, an IMEI (International Mobile Equipment Identity)).

[0143] Note that the application ID may or may not be replaced with a "user ID." Furthermore, if the application allows only one account to be registered per terminal 20, for example, the "identification information for identifying the terminal 20 = identification information for identifying the user of the terminal 20 = application ID" may be used.

[0144] Also, for example, it may or may not be possible to assign multiple terminal IDs to one application ID. In this case, it may or may not be possible to simultaneously launch an application on multiple terminals 20 using one application ID as an identification (login) target.

[0145] It is also possible to apply a method of managing accounts using information such as a terminal telephone number instead of various IDs such as an application ID. In this case, instead of storing information on IDs such as an application ID in the account registration data 153, information on terminal telephone numbers can be stored in the account registration data 153. It is also possible to have one-to-one correspondence between information on IDs such as application IDs and information on terminal telephone numbers, without replacing information on IDs such as application IDs with information on terminal telephone numbers, but this is not necessary.

[0146] In the following examples, for the sake of simplicity, it is assumed that one account is registered for one terminal 20. In this case, as described above, "identification information for identifying the terminal 20 = identification information for identifying the user of the terminal 20 = application ID," and therefore, the term "user of the account" used in the following description may or may not be substantially synonymous with "terminal of the account."

[0147] (2) Functional Configuration of the Terminal Fig. 2-5 is a diagram showing an example of functions realized by the control unit 21 of the terminal 20 in this embodiment. The control unit 21 includes, as a functional unit, an application processing unit 211 for executing application processing in accordance with an application processing program 281 stored in the storage unit 28, for example.

[0148] 2-6 is a diagram showing an example of data stored in the storage unit 28 of the terminal 20 in this embodiment. The storage unit 28 stores, for example, an application processing program 281 to be executed as application processing, and an application ID 283 corresponding to the terminal 20 or the account of the user of the terminal 20.

[0149] <Display Screen> Display screen examples will be described below. The transition of the display screens described below is merely an example of the transition of the display screens for realizing the method of the present disclosure. In the transition of the display screens exemplified below, the display of some display screens may be omitted, or other display screens may be added. Furthermore, the terms used in the display screens described below may differ from the terms used in the processes described thereafter (some terms may not be consistent).

[0150] In the following, the terminal 20 is illustrated as an example in which the terminal 20 is a smartphone equipped with a display unit 24 having a vertically long display. For example, the smartphone has a touch panel that functions as an input unit, which is disposed opposite the display, constituting a touch screen. When an element such as an icon, button, item, or input area is displayed on the display, and a part of the touch panel that faces the area where the element is displayed is operated by a user, a program associated with the element or a subroutine of the program may be executed.

[0151] Note that the flow of screen transitions may be explained by showing predetermined screens displayed on different terminals 20 within one drawing or across multiple drawings. In this case, for example, a predetermined screen may be displayed on a different terminal 20 based on (triggered by) at least one of the following: - Automatically displayed - Displayed when some kind of notification (including push notification, etc.; the same applies below) is sent to the terminal 20 and the user performs an operation in response to the notification - No notification is sent to the terminal 20, but the user performs a predetermined operation on an application - Displayed when some kind of notification is sent to the terminal 20 and the user performs a predetermined operation on the application that sent the notification

[0152] In the following, an example is given in which the terminal 20A of user A.A. is used as an example of an account that generates an adversarial image. The terminal 20A may be used as an example of a first terminal, and the user A.A. may be used as an example of a first user of the first terminal. The terminal 20A or the account of the user A.A. may be used as an example of a first account. In this example, the generation of the adversarial image is performed within an adversarial image generation application, and the name of the adversarial image generation application is denoted as "Image Protection App." Note that the generation of the adversarial image may be performed on, for example, a messaging application different from the adversarial image generation application. In this case, for example, an original image and a watermark image may be acquired based on a conversation in a chat room of the messaging application, and the adversarial image may be presented.

[0153] 2-7 is a diagram showing an example of a screen displayed on the display unit 24 of the terminal 20A in this embodiment. In the drawings, what is referred to as a "base image" will be referred to as an "original image" where appropriate. Furthermore, what is referred to as a "digital watermark" will be referred to as a "watermark image" where appropriate. Furthermore, what is referred to as a "protection image" will be referred to as an "adversarial image" where appropriate.

[0154] 2-7 shows an example of an original image selection screen of the adversarial image generation application displayed on the display unit 24 of the terminal 20A. On this screen, for example, four images are selected as original images from images stored in an image database (not shown) stored in the storage unit 28 of the terminal 20A.

[0155] When the original image is selected, the display changes to, for example, a watermark image selection screen in the center. On the watermark image selection screen, for example, an image to be applied to the image selected as the original image (in this screen, an image including the character string "Genuine A.A." as identification information) is selected from among the watermark images created in advance by the user of terminal 20. Note that, for example, this screen may be configured to allow a new watermark image to be created. Note that selecting a watermark image may also be considered as selecting (or inputting or setting) the identification information included in the watermark image.

[0156] When a watermark image is selected, for example, the selected original image and the watermark image are transmitted to the server 10, and the perturbation control unit optimization process and the adversarial image output process are executed in the server 10. Then, the generated adversarial image is transmitted to the terminal 20A.

[0157] Then, for example, the display changes to the hostile image confirmation screen on the right. This screen displays a list of the generated hostile images. This screen also displays a "Save Selected Image" button for saving an image selected by the user from the hostile images to the terminal 20, and a "Save All Images" button for saving all hostile images at once.

[0158] In addition, the hostile image confirmation screen may be configured to allow the hostile image to be shared (posted) on any SNS service, messaging service, image sharing service, or the like.

[0159] 2-8 is a flowchart showing an example of the flow of processing executed by each device in this embodiment. From the left, this figure shows an example of processing executed by the control unit 21 of user A's terminal 20A, and an example of processing executed by the control unit 11 of the server 10.

[0160] First, the control unit 21 of the terminal 20A selects and acquires original image information including one or more original images based on, for example, an input to the input / output unit 23 of the terminal 20A (for example, a user input (such as an operation input or sound input by the user); the same applies below). The original images may be captured by, for example, the imaging unit 27 of the terminal 20A. Then, the control unit 21 of the terminal 20A transmits the original image information to the server 10 via the communication I / F 22 (the same applies below) (A110).

[0161] Furthermore, when the control unit 21 of the terminal 20A selects and acquires watermark image information including a watermark image based on, for example, a user input, the control unit 21 transmits the watermark image information to the server 10 (A120).

[0162] When the original image information and watermark image information are received from the terminal 20A via the communication I / F 14 (same applies hereinafter), the server 10 refers to the account registration data 153, associates the received watermark image information with the application ID of the terminal 20A, and stores the information. Then, the control unit 11 of the server 10 executes a perturbation control unit optimization process (S110). The perturbation control unit optimization process is executed, for example, in accordance with steps P130 to P200 in FIG. 1-2.

[0163] The control unit 11 of the server 10 may refer to the account registration data 153, and if watermark image information identical to or similar to the received watermark image information (e.g., the correlation between images is higher than a threshold) is already registered, may transmit watermark image resend request information to the terminal 20A to request resend of watermark image information including a different watermark image. Upon receiving the watermark image resend request information from the server 10, the control unit 21 of the terminal 20A may output (e.g., display) the received watermark image resend request information, and may transmit the watermark image information to the server 10 again based on a user input.

[0164] Then, the control unit 11 of the server 10 executes the adversarial image generation process (S120). The adversarial image generation process is executed, for example, in accordance with step P140 in FIG. 1-2 using the optimized (post-learning) model weight “G” of the perturbation control unit 2.

[0165] Then, the control unit 11 of the server 10 transmits hostile image information including the generated hostile image to the terminal 20A (S130).

[0166] The control unit 11 of the server 10 may store, for example, the model weight "G" of the perturbation control unit 2 optimized in the perturbation control unit optimization process, the original image information, and the watermark image information in association with each other in the account registration data 153. Then, the control unit 11 may transmit optimized model weight list information, which is a list of the stored model weights of the perturbation control unit 2, to the terminal 20. The optimized model weight list information may include the original image and the watermark image used in the perturbation control unit optimization process.

[0167] Upon receiving the optimized model weight list information, the control unit 21 of the terminal 20 may, for example, display the list information. Then, model weight designation information for designating model weights to be used for adversarial image generation, among the optimized model weight list information, may be transmitted to the server 10. The control unit 11 of the server 10 may execute an adversarial image generation process using the model weight “G” designated by the model weight designation information, the original image information, and the watermark image information, and transmit adversarial image information including the generated adversarial image to the terminal 20. In this way, when using the optimized original image information and watermark image information, an adversarial image may be generated without performing the perturbation control unit optimization process again.

[0168] The control unit 11 of the server 10 determines whether to terminate the process (S190). If it is determined that the process should be continued (S190: NO), the control unit 11 of the server 10 waits for reception of original image information, for example, and returns the process to S110. On the other hand, if it is determined that the process should be terminated (S190: YES), the control unit 11 of the server 10 terminates the process.

[0169] When receiving the hostile image information from the server 10, the control unit 21 of the terminal 20A outputs the received hostile image information (for example, displays it on the display unit 24). Note that the control unit 21 of the terminal 20A may store the received hostile image information in the storage unit 28.

[0170] This allows the user of terminal 20A to easily obtain an adversarial image that is comparable in appearance to the original image but that, when used as a reference image by a diffusion model, produces a watermark image. Furthermore, the perturbation control unit optimization process may be difficult on a portable terminal such as a smartphone due to issues with processor processing power (e.g., the number of parallel processes), resources (e.g., the amount of memory used for calculations), power consumption, etc. However, by executing the perturbation control unit optimization process on server 10, which has ample processing power and allowable power consumption, the perturbation control unit optimization process can be realized within a practical processing time, regardless of the processing power, etc., of terminal 20.

[0171] Then, the control unit 21 of the terminal 20A determines whether to end the process (A190). If it is determined that the process should be continued (A190: NO), the control unit 21 of the terminal 20A returns the process to A110, for example. On the other hand, if it is determined that the process should be ended (A190: YES), the control unit 21 of the terminal 20A ends the process.

[0172] <Effects of the Second Example> According to this example, an author or the like uses his / her terminal 20 to select an original image that he / she wishes to protect and a watermark image that he / she wishes to use, and an adversarial image based on the selected original image and watermark image is generated by the server 10 and provided to the terminal 20. The author or the like can easily obtain an adversarial image based on the original image that he / she wishes to protect. Furthermore, as will be described in the examples below, by registering the adversarial image thus obtained in the server 10, it is possible to prevent the adversarial image based on the original image from being used as a reference image for a diffusion model or for additional training of the diffusion model.

[0173] <Second Modification (1)> In the above embodiment, the perturbation control unit optimization process, the adversarial image generation process, and communication with the terminal 20 are performed in the server 10, but this is not limiting. For example, the server 10 may be a server system configured with a front-end server 10F that provides a messaging service and an image transmission / reception interface service, and a back-end server 10B that provides the perturbation control unit optimization process and the adversarial image generation process.

[0174] The processing in this case will be exemplified below. For example, the front-end server 10F receives the original image information and the watermark image information from the terminal 20. Then, the front-end server 10F refers to the account registration data 153 and registers the watermark image information.

[0175] The front-end server 10F then transmits adversarial image generation request information, including the original image information and the watermark image information, to the back-end server 10B. The back-end server 10B executes a perturbation control unit optimization process and an adversarial image generation process based on the received adversarial image generation request information. Then, the back-end server 10B transmits adversarial image information, including the generated adversarial image, to the front-end server 10F.

[0176] When the front-end server 10F receives the hostile image information, it transmits it to the terminal 20.

[0177] <Second Modification (2)> In the above embodiment, the server 10 transmits an adversarial image to the terminal 20 when it executes the perturbation control unit optimization process, but this is not limiting. For example, the server 10 may transmit perturbation control unit model weight information including the model weight "G" of the optimized perturbation control unit 2 to the terminal 20 when it executes the perturbation control unit optimization process.

[0178] Upon receiving the perturbation control unit model weight information, the terminal 20 may store, for example, the optimized model weight “G”, the original image information, and the watermark image information in the storage unit 28 .

[0179] Then, when generating an adversarial image based on the same original image information and watermark image information again, the server 10 may transmit the model weight “G” to the server 10 in addition to the original image information and the watermark image information. When the server 10 receives the model weight “G” from the terminal 20, the server 10 may omit the perturbation control unit optimization process and execute the adversarial image generation process based on the received model weight “G”.

[0180] The terminal 20 may be configured to execute the adversarial image generation process upon receiving the optimized model weight "G" from the server 10. The adversarial image generation process does not require the operation of the diffusion model unit 3. Therefore, by implementing only the perturbation control unit 2, which has a relatively simple structure compared to the diffusion model unit 3, in the terminal 20 and executing the adversarial image generation process, the load on the server 10 can be reduced. Furthermore, the user of the terminal 20 can generate and acquire adversarial images at any time, regardless of the operating status or communication situation of the server 10.

[0181] <Third Example> In the third example, when a second user generates a diffusion-generated image using a hostile image generated by a first user as a reference image, the server 10 notifies the second user of information about the first user who generated the hostile image (e.g., the author of the original image) based on a watermark image embedded in the hostile image.

[0182] The contents described in the third embodiment can be similarly applied to any of the other embodiments and other modified examples.

[0183] <Display Screen> In this example, a service that generates a diffusion image based on a diffusion model using a reference image is referred to as a diffusion image generation service, and an application for realizing the diffusion image generation service is referred to as a "diffusion image generation application." The name of the diffusion image generation application is exemplified as "Image Creation App." The diffusion image generation application may allow any user to register a reference image.

[0184] In the following, a case will be exemplified in which user A.A's terminal 20A is an example of an account that registers an adversarial image generated in the adversarial image generation service as an example of a reference image in the diffusion image generation service. Also, a case will be exemplified in which user B.B's terminal 20B is an example of an account that generates a diffusion-generated image using the adversarial image as a reference image in the diffusion image generation service.

[0185] The diffusion image generation service and the adversarial image generation service may be an integrated service or may be separate services. The diffusion models used in the diffusion image generation service and the adversarial image generation service may have common model weights. Furthermore, the diffusion model used in the diffusion image generation service may be a diffusion model that has undergone fine tuning or additional learning using the diffusion model used in the adversarial image generation service as a pre-trained model. In this example, for example, the diffusion image generation service and the adversarial image generation service are separate services (different applications) and the accounts for each service are linked.

[0186] 3-1 and 3-2 are diagrams showing examples of screens displayed on the display unit 24 of each terminal 20 in this embodiment. In the drawings, what is referred to as a "reference image" will be explained as a "reference image" as appropriate. Furthermore, what is referred to as a "protected image" in the drawings will be explained as a "hostile image" as appropriate.

[0187] The left side of FIG. 3-1 shows an example of a reference selection screen of the diffusion image generation application displayed on the display unit 24 of the terminal 20B. On this screen, for example, one image is selected as a reference image from images stored in a reference image database (not shown) stored in the storage unit 28 of the server 10. The reference image database may contain adversarial images. In this example, the selected reference image is the adversarial image generated in FIG. 2-7.

[0188] The reference image may be selected from images stored in an image database (not shown) stored in the storage unit 28 of the terminal 20B. The image database may store images acquired by the user B.B. from a social networking service, an image sharing service, or the like. In this case, the image database of the terminal 20B may include hostile images.

[0189] When a reference image is selected, the display changes to, for example, a prompt input screen on the right. On the prompt input screen, an image generation prompt is input in the prompt input region PTR, and when the "Generate" button at the bottom of the screen is tapped, a diffusion generated image generation process is executed, for example, in the diffusion model unit 3 of the server 10. Then, a diffusion generated image is generated based on the reference image and the input prompt. The generated diffusion generated image is transmitted to terminal 20B.

[0190] Then, for example, the display changes to the diffuse generated image confirmation screen shown on the left side of FIG. 3-2. On this screen, the generated diffuse generated image includes the character string "Genuine A.A." (appears as a watermark). This allows user B.B., who generated the diffuse generated image, to realize that he or she used an adversarial image as a reference image.

[0191] Additionally, below the diffusely generated image, hostile image use warning information is displayed to notify the user that a hostile image was used as a reference image. The hostile image use warning information displays, for example, information about user A.A., who generated and registered the hostile image. This allows user B.B., who generated the diffusely generated image, to be notified that user A.A. is the contact for image usage permission, etc.

[0192] Furthermore, for example, if an adversarial image is used as a reference image, a notification may be sent to the adversarial image generation application. An example of output of adversarial image use warning information in the adversarial image generation application is shown on the right. This screen displays information about user B.B., who attempted to generate a diffusely generated image using an adversarial image as a reference image. Note that the adversarial image use warning information may include, for example, the adversarial image used as a reference image.

[0193] <Processing> Figure 3-3 is a flowchart showing an example of the flow of processing executed by each device in this embodiment. From the left, this figure shows an example of processing executed by the control unit 21 of user A's terminal 20A, an example of processing executed by the control unit 11 of the server 10, and an example of processing executed by the control unit 21 of user B's terminal 20B. Note that in this example, the server 10 may provide both the hostile image generation service and the diffuse image generation service.

[0194] First, the control unit 21 of the terminal 20A transmits, to the server 10, hostile image registration information for enabling the hostile image generated in the hostile image generation service to be used as a reference image in the diffusion image generation service, for example, based on a user input (A210). The hostile image registration information may include one or more hostile images.

[0195] When the hostile image registration information is received from the terminal 20A, the control unit 11 of the server 10 executes a hostile image registration process (S210). In the hostile image registration process, the control unit 11 of the server 10 stores a hostile image in a reference image database based on the received hostile image registration information, for example.

[0196] The control unit 21 of the terminal 20B transmits reference image list request information to the server 10 based on, for example, a user input, for obtaining a list of images to be used as reference images in the diffusion image generation service (B210). Then, the control unit 11 of the server 10 refers to the reference image database and transmits reference image list information, which is a list of registered reference images, to the terminal 20B (S220).

[0197] For example, when an image to be used as a reference image and a prompt are received from the reference image list information based on a user input, the control unit 21 of the terminal 20B transmits diffusion image generation request information including the reference image and the prompt to the server 10 (B220). Note that the diffusion image generation request information may not include the prompt. Furthermore, the reference image may be, for example, an image stored in the storage unit 28 of the terminal 20B.

[0198] When receiving the diffusion generated image generation request information from terminal 20B, the control unit 11 of the server 10 executes a diffusion generated image generation process (S230). In the diffusion generated image generation process, the control unit 11 of the server 10 inputs a reference image and a prompt to the diffusion model unit 3, for example, to generate a diffusion generated image. Note that in the diffusion generated image generation process, the control unit 11 of the server 10 may input a reference image to the diffusion model unit 3 to generate a diffusion generated image.

[0199] Then, the control unit 11 of the server 10 transmits the diffused generated image information including the generated diffused generated image to the terminal 20B (S240).

[0200] When the diffusion-generated image information is received from the server 10, the control unit 21 of the terminal 20B outputs (for example, displays) the received diffusion-generated image information (B230). The control unit 21 of the terminal 20B may store the diffusion-generated image included in the received diffusion-generated image information in the storage unit 28.

[0201] Then, the control unit 11 of the server 10 executes a watermark image information detection process (S250). In the watermark image information detection process, the control unit 11 of the server 10, for example, refers to the account registration data 153 and determines whether the diffusion-generated image includes each image registered as watermark image information. The determination of whether watermark image information is included may be made based on, for example, the correlation between images. Alternatively, the determination of whether watermark image information is included may be made using, for example, an API provided by a watermark detection server (not shown).

[0202] When it is determined that a watermark image has been detected (S250: YES), the control unit 11 of the server 10 transmits, for example, hostile image use warning information indicating that a hostile image has been used as a reference image to the terminal 20A and the terminal 20B (S260).When it is determined that a watermark image has not been detected (S250: NO), the control unit 11 of the server 10 terminates the process, for example.

[0203] When it is determined that the hostile image use warning information has been received from the server 10 (A220: YES), the control unit 21 of the terminal 20A, for example, outputs (e.g., displays) the received hostile image use warning information (A230). When it is determined that the hostile image use warning information has not been received from the server 10 (A220: NO), the control unit 21 of the terminal 20A, for example, terminates the processing.

[0204] The same applies to the terminal 20B.

[0205] In the system disclosed in this specification, even if it is determined whether a watermark image is detected in a diffusion-generated image after the diffusion-generated image is transmitted to the creator's terminal 20, no problems (e.g., copyright infringement) will occur because the watermark image is embedded in the diffusion-generated image itself in a state visible to the user.

[0206] <Effects of the Third Example> According to the present example, when an adversarial image is used as a reference image for a diffusion model to generate a diffusion-generated image, or when an adversarial image is used for additional learning of a diffusion model and a diffusion-generated image is generated using the diffusion model after the additional learning, warning information is sent to the terminal of the user who is the author of the original image or the terminal of the user who attempted to generate the diffusion-generated image, thereby making it possible to prevent the adversarial image from being used for purposes not intended by the author of the original image, etc.

[0207] <Third Modification (1)> In the above embodiment, the hostile image generation service and the diffuse image generation service are provided by the server 10, but this is not limiting. For example, the server 10A that provides the hostile image generation service and the server 10D that provides the diffuse image generation service may be separated.

[0208] For example, the server 10A provides the terminal 20 with an adversarial image based on the original image and the watermark image according to FIG. 2-8. Furthermore, for example, the server 10D provides the terminal 20 with a diffusion-generated image based on the adversarial image according to FIG. 3-3. The server 10D may have a diffusion model configured with a model weight "θ" similar to that of the diffusion model unit 3 in the server 10A, or a diffusion model configured with a model weight "θ'" obtained by fine-tuning or additional learning the diffusion model unit 3. The server 10D does not require the perturbation control unit 2.

[0209] 3C, for example, the terminal 20A may transmit the hostile image and the watermark image to the server 10D. Then, the server 10D may store the watermark image information in the storage unit 15 in association with the account information of the terminal 20A.

[0210] In the information processing device 1 disclosed in this specification, there is no need to change (learn) the model weight “θ” of the diffusion model unit 3 in order to generate an adversarial image. This allows the servers 10 that provide services to be flexibly separated.

[0211] <Third Modification (2)> In the above embodiment, the adversarial image was used based on the aforementioned diffuse image generation method (A), but this is not limiting. For example, the adversarial image may be used for additional learning, etc., based on the diffuse image generation method (B). In this case, for example, if a prompt associated with the style of the adversarial image is used in a diffusion model that has learned the style of the adversarial image, a watermark image is embedded in the diffuse image. Therefore, for example, if an image similar to the style of the adversarial image is generated in the learned diffusion model, the server 10 can transmit adversarial image use warning information to the terminal 20.

[0212] <Third Modification (3)> In the above embodiment, when a watermark image is detected in a diffusion-generated image, hostile image use warning information is sent to terminal 20A that generated a hostile image and made it usable as a reference image, and terminal 20B that generated a diffusion-generated image using the hostile image as a reference image, but this is not limited to this.

[0213] Figure 3-4 shows a list of various patterns that can be set as destinations for the hostile image use warning information. In pattern (A), for example, the information is sent to a user who has registered an adversarial image as a reference image in the viral image generation service. In pattern (B), for example, the information is sent to a user who has generated an adversarial image from an original image in the adversarial image generation service. In pattern (C), for example, the information is sent to a user who has generated a diffuse-generated image using an adversarial image as a reference image in the viral image generation service. In pattern (D), for example, the information is sent to a user who has performed additional training on a viral model based on adversarial image information in the viral image generation service. Pattern (E) and subsequent patterns are any combination of patterns (A) to (D).

[0214] This allows the server 10 to set the destination of the hostile image use warning information within an appropriate range.

[0215] <Third Modification (4)> In the above embodiment, when a diffusion-generated image is generated using an adversarial image, a diffusion-generated image with an embedded watermark image is generated, but this is not limited to this. For example, after a diffusion-generated image with an embedded watermark image is generated, if a user who generated a diffusion-generated image using the adversarial image as a reference image in a diffusion image generation service obtains permission to use the original image from a user who registered the adversarial image as a reference image in the diffusion image generation service or a user who generated an adversarial image from an original image in the adversarial image generation service, the reference image may be switched to the original image in the diffusion image generation service and a diffusion-generated image may be generated again.

[0216] In this case, in FIG. 3-3, for example, the hostile image registration information transmitted from the terminal 20A may include the hostile image and the original image. Then, the server 10 may store the hostile image and the original image in the reference image database. Of the two images, only the hostile image may be included in the reference image list information.

[0217] For example, when terminal 20B displays the hostile image use warning information, terminal 20B may transmit original image use permission application information to server 10 based on a user input. Then, server 10 may transmit original image use permission request information to terminal 20A.

[0218] Terminal 20A may display the received original image license request information and transmit the original image license information based on a user input to server 10. Upon receiving the original image license information, server 10 may execute the diffusion generated image generation process again using the original image as a reference image and transmit diffusion generated image information without an embedded watermark image to terminal 20B.

[0219] This allows the user of terminal 20B to generate a diffusion generated image and confirm whether the generated image matches the user's intention before obtaining permission to use from the user of terminal 20A. After confirming that the diffusion generated image matches the user's intention, the user of terminal 20B can apply for permission to use from the user of terminal 20A and obtain the diffusion generated image without a watermark.

[0220] Furthermore, the user of terminal 20A can spread the impression of a diffusion-generated image based on a hostile image by publishing the hostile image without issuing a license. By releasing the original image as a reference image only to users who request a license, the user can avoid the need to request a license each time. This allows users to easily and widely publish their own works while effectively preventing copyright infringement.

[0221] <Others>

[0222] At least a part of the processing that is to be performed by the server 10 in the above embodiment may be performed by the terminal 20. Conversely, at least a part of the processing that is to be performed by the terminal 20 in the above example may be performed by the server 10.

[0223] The operator of the server 10 may also be a provider of an adversarial image generation service or a viral image generation service in partnership with a messaging service provider. In this case, the processes described in the above embodiments may be realized by a single server, or the processes described in the above embodiments may be shared and realized by a server system composed of multiple servers. Note that a system composed of one or more servers may be defined as a server system, and the server of the present invention may be considered as a server system.

[0224] Furthermore, in the above-described embodiments, the server for distributing various applications (the server from which the terminal 20 downloads the applications) may be configured as a server different from the server that provides the corresponding service (application). In other words, the server for distributing applications and the server that performs the application management process described in the above-described embodiments may be configured as physically separated servers, or may be configured as a single server.

[0225] Furthermore, applications are not limited to various application programs, but may also include, for example, a program that provides a function of another service as one function of a base application (for example, a program that provides a function of an adversarial image generation service or a viral image generation service as one function of a messaging application, or vice versa), a program for updating the base application, etc. Also, applications may include data used in application programs (which may include data for updating applications, etc.).

[0226] Furthermore, in the above embodiments, the present invention has been described as being implemented using a client-server system, but is not limited to this. As mentioned above, the present invention may be implemented using a system such as a distributed system in which the functions of a server or server system are provided in the terminal 20. For example, the processes described in the flowcharts of the above embodiments as being performed by a server or server system may be performed by a terminal.

[0227] Furthermore, as mentioned above, the contents described in the above-mentioned embodiments, modifications, other embodiments, etc. can be applied in combination with each other.

[0228] REFERENCE SIGNS LIST 1 Information processing device 100 Communication system 10 Server 20 Terminal 30 Network

Claims

1. An information processing device capable of generating an adversarial image in a diffusion model, comprising: a control unit that generates the adversarial image based on a first image and a watermark image including identification information; and the adversarial image is an image in which, when a second image is generated based on the diffusion model and the adversarial image, the identification information is visible in the second image.

2. An information processing device according to claim 1, wherein the identification information includes information that can identify a user who has the authority to use the first image as input to the diffusion model or the authority to use the first image for additional learning of the diffusion model.

3. An information processing device according to claim 2, wherein the identification information includes one or more of a character string, a signature, a code, and a fingerprint.

4. An information processing device according to claim 1, wherein the control unit includes a noise generation unit that generates noise based on the first image and the watermark image, and generates the hostile image based on the noise and the first image.

5. An information processing device according to claim 4, wherein the control unit calculates a first constraint condition relating to the first image and the adversarial image, and calculates a second constraint condition relating to the watermark image and the second image, and the noise generation unit is optimized based on the first constraint condition and the second constraint condition.

6. An information processing device according to claim 5, wherein the noise generating section is optimized based on weighting relating to the first constraint condition and the second constraint condition.

7. An information processing device according to any one of claims 1 to 6, wherein the adversarial image is an image in which, when the second image is generated based on inputting the adversarial image into the diffusion model, the identification information becomes visible in the second image.

8. An information processing device according to any one of claims 1 to 6, wherein the adversarial image is an image in which the identification information becomes visible in the second image when the second image is generated based on the diffusion model in which the adversarial image was used for additional learning.

9. An information processing device according to any one of claims 1 to 6, wherein the adversarial image is an image in which, when the second image is generated based on inputting the adversarial image into the diffusion model, the identifying information is visible in the second image, and when the third image is generated based on the diffusion model in which the adversarial image is used for additional learning, the identifying information is visible in the third image.

10. A server communicating with at least a first terminal, wherein a control unit of the server includes the information processing device described in claim 1, and the control unit of the server receives first information regarding the first image and the watermark image from the first terminal via a communication unit of the server, performs an association process based on a first account of the first terminal and the watermark image, generates the hostile image, and transmits second information regarding the hostile image to the first terminal via the communication unit.

11. A server as described in claim 10, wherein a control unit of the server receives the hostile image from a second terminal via a communication unit of the server, generates the second image, transmits the second image to the second terminal via the communication unit, and, if the second image includes the watermark image associated with the first account, transmits information about the first account to the second terminal.

12. An information processing method in an information processing device capable of generating an adversarial image in a diffusion model, the adversarial image being generated based on a first image and a watermark image including identification information, the adversarial image being an image in which, when a second image is generated based on the diffusion model and the adversarial image, the identification information is visible in the second image.

13. A program executed by an information processing device capable of generating an adversarial image in a diffusion model, the program generating the adversarial image by a control unit of the information processing device based on a first image and a watermark image including identification information, and the adversarial image is an image in which, when a second image is generated based on the diffusion model and the adversarial image, the identification information becomes visible in the second image.

Images