Information processing device, information processing method, and information processing program

Abstract

To appropriately generate other information using information when an image is generated using an image generation model.SOLUTION: An information processing device comprises an acquisition unit and a generation unit. The acquisition unit acquires a diffusion model, which is a model learned so as to restore an image indicated by a text step by step from a noise image and also learned so as to restore an image according to an attention based on the text, and an input text entered by a user. The generation unit generates a corresponding image corresponding to the input text using the diffusion model and input text acquired by the acquisition unit, and generates mask information within the target range indicated by the input text in the corresponding image on the basis of the attention used by the diffusion model when the corresponding image was generated.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and an information processing program. [Background technology]

[0002] Conventionally, there have been provided techniques for generating various types of information related to an image, such as a technique for identifying the background in an image and generating information such as mask information (mask image) that masks that portion. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-014263 [Patent Document 2] Japanese Patent Publication No. 2023-030850 [Non-patent literature]

[0004] [Non-Patent Document 1] “Prompt-to-Prompt Image Editing with Cross Attention Control”, Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or<Internet>https: / / arxiv.org / pdf / 2208.01626.pdf (retrieved March 10, 2020) [Non-patent document 2] “Denoising Diffusion Probabilistic Models”, Jonathan Ho, Ajay Jain, and Pieter Abbeel, <Internet> https: / / arxiv.org / pdf / 2006.11239.pdf (Retrieved March 10, 2023) [Non-patent document 3] “High-Resolution Image Synthesis with Latent Diffusion Models”, Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjorn Ommer, <Internet> https: / / arxiv.org / pdf / 2112.10752.pdf (Retrieved March 10, 2023) [Non-patent document 4] “Null-text Inversion for Editing Real Images using Guided Diffusion Models”, Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or, <Internet> https: / / arxiv.org / pdf / 2211.09794.pdf (Retrieved March 10, 2023) Summary of the Invention [Problem to be solved by the invention]

[0005] However, there is room for improvement in the above-mentioned conventional techniques. For example, the above-mentioned conventional techniques require a dedicated model for identifying the background and generating mask information. Without such a dedicated model, it is difficult to generate the desired information. Therefore, it is desirable to appropriately generate other information, such as mask information, using information obtained during processing in a model that generates an image.

[0006] The present application has been made in consideration of the above, and aims to provide an information processing device, an information processing method, and an information processing program that appropriately generate other information using information obtained when an image is generated using a model that generates the image. [Means for solving the problem]

[0007] The information processing device of the present application is characterized by comprising: a diffusion model, which is a model that has been trained to gradually restore an image indicated by text from a noise image and has been trained to restore the image in accordance with attention based on the text; an acquisition unit that acquires input text entered by a user; and a generation unit that uses the diffusion model and the input text acquired by the acquisition unit to generate a corresponding image corresponding to the input text, and generates mask information of the corresponding image for a target range indicated by the input text based on the attention used by the diffusion model when generating the corresponding image. [Effects of the Invention]

[0008] According to one aspect of the embodiment, it is possible to provide an effect that other information can be appropriately generated using information obtained when an image is generated using a model that generates the image. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating an example of information processing according to an embodiment. [Figure 2] FIG. 2 is a diagram illustrating another example of information processing according to the embodiment. [Figure 3] FIG. 3 is a diagram illustrating an example of the configuration of an information processing system according to the embodiment. [Figure 4] FIG. 4 is a diagram illustrating an example of the configuration of the information processing device according to the embodiment. [Figure 5] FIG. 5 is a diagram illustrating an example of a model information storage unit according to the embodiment. [Figure 6] FIG. 6 is a flowchart illustrating an example of information processing according to the embodiment. [Figure 7] FIG. 7 is a flowchart illustrating an example of information processing according to the embodiment. [Figure 8] FIG. 8 is a flowchart illustrating an example of information processing according to the embodiment. [Figure 9] FIG. 9 is a diagram illustrating an example of a model configuration. [Figure 10]FIG. 10 is a diagram illustrating an example of the configuration of the main part of the model. [Figure 11] FIG. 11 is a diagram illustrating an example of a hardware configuration. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, a detailed description will be given of an information processing device, an information processing method, and an information processing program (hereinafter referred to as an "embodiment") according to the present application, with reference to the drawings. Note that the information processing device, the information processing method, and the information processing program according to the present application are not limited to the embodiment. Furthermore, the same components in the following embodiments are denoted by the same reference numerals, and redundant description will be omitted.

[0011] (Embodiment) [1. Information Processing] An example of information processing according to the embodiment will be described below with reference to FIG. 1. FIG. 1 is a diagram illustrating an example of information processing according to the embodiment. Specifically, FIG. 1 illustrates an example of information processing including a model learning process according to the embodiment and an image generation process using the model. FIG. 1 illustrates an example in which an information processing device 100 (see FIG. 4) learns (generates) a diffusion model M1 (hereinafter also referred to as "model M1"), which is a diffusion model for generating an image from a noise image, and generates an image using the generated model M1. Furthermore, the point that other information such as mask information is generated using information obtained during the image generation process using model M1 will be described after the overview of the model is explained in FIG. 1.

[0012] [1-1. Learning process example] First, an example of the learning process of the model M1 will be described with reference to Fig. 1. First, data (learning data) used in the learning process of the model M1 will be described.

[0013] 1, the information processing device 100 acquires data (learning data) used in the learning process of the model M1. For example, the information processing device 100 acquires the learning data of the model M1 from a learning data storage unit 121 (see FIG. 4). Note that, if the learning data is not registered in the learning data storage unit 121, the information processing device 100 may request an external device such as the information providing device 50 (see FIG. 3) to provide the learning data, and register the data received from the external device in the learning data storage unit 121 as learning data (also referred to as "learning data").

[0014] 1 shows an image group IG1 including a plurality of staged images in which noise is added to a predetermined image in stages, as an example of data used in the learning process of the model M1. The information processing device 100 learns the model M1 using learning data including the image group IG1 shown in FIG.

[0015] The image group IG1 includes an image P10, which is an image (learning image) of a predetermined object. The image P10 is an image of a sofa and a person (woman) as the predetermined object. The image P10 is at level #0 in terms of noise addition, i.e., it is an image with no noise added.

[0016] The image group IG1 also includes an image P11, which is an image obtained by adding noise to the image P10. The image P11 is an image obtained by adding one level of noise to the image P10. The level of noise addition for the image P11 is level #1, i.e., the image is an image to which one level of noise has been added. For example, the image P11 is generated by a noise addition process (process SN1 in FIG. 1) that adds noise to the image P10.

[0017] The image group IG1 also includes an image P12, which is an image obtained by adding noise to the image P11. The image P12 is an image obtained by adding another level of noise to the image P11. The level of noise addition for the image P12 is level #2, i.e., the image is an image to which two levels of noise have been added. For example, the image P12 is generated by a noise addition process (process SN2 in FIG. 1) that adds noise to the image P11.

[0018] The image group IG1 also includes an image P13, which is an image obtained by adding noise to the image P12. For example, a noise adding process (process SN3 in FIG. 1) adds noise to the image P12 to generate an image (image P13) at stage #3, where three stages of noise have been added. Note that because the noise is repeated, a description of the intermediate processes will be omitted.

[0019] The image group IG1 also includes an image P1K-2, which is an image obtained by adding noise to an image P1K-3 to which K-3 levels of noise have been added, where K is an arbitrary number (value), for example, 30, 1000, etc. For example, a noise adding process (process SNK-2 in FIG. 1) that adds noise to the image P1K-3 generates an image (image P1K-2) at level #K-2 to which K-2 levels of noise have been added.

[0020] Furthermore, the image group IG1 includes an image P1K-1, which is an image obtained by adding noise to the image P1K-2. The image P1K-1 is an image obtained by adding another level of noise to the image P1K-2. The level of noise addition for the image P1K-1 is level #K-1, i.e., the image is an image to which K-1 levels of noise have been added. For example, the image P1K-1 is generated by a noise addition process (process SNK-1 in FIG. 1) that adds noise to the image P1K-2.

[0021] The image group IG1 also includes an image P1K, which is a noise image. For example, the image P1K is an image in which one more level of noise has been added to the image P1K-1. The level of noise addition for the image P1K is level #K, i.e., the image is an image to which K levels of noise have been added. For example, the image P1K is generated by a noise addition process (process SNK in FIG. 1) that adds noise to the image P1K-1. Processes SN1 to SNK in FIG. 1 correspond to a diffusion process (forward process) in a diffusion model.

[0022] For example, each image in the image group IG1 may be generated by a process (noise-adding process) in which Gaussian noise is added to data to gradually transition from actual data to complete Gaussian noise. For example, when K is 1000, each image in the image group IG1 is generated by mixing data and noise while changing the ratio of data to noise so that complete noise is achieved after 1000 noise-adding processes. Note that the case where K is 1000, i.e., the number of noise-adding processes is 1000, is merely an example, and the number of noise-adding processes may be less than 1000, such as 32 times or 1 time, or may be more than 1000 times. Furthermore, each image in the image group IG1 may be generated by the information processing device 100 or an external device such as the information providing device 50.

[0023] As such, the image group IG1 shown in FIG. 1 includes a plurality of images such as images P10, P11, P12, ... P1K-2, P1K-1, P1K, etc. Below, an example will be described in which the information processing device 100 learns the model M1 using the image group IG1 described above. Note that the above is merely an example, and the images included in the image group IG1 may be any images. For example, the images included in the image group IG1 are not limited to real images such as captured images (photographs) of a subject (object) actually captured by an imaging device such as a camera, but may also be images generated by information processing using an image generation model, etc. Furthermore, for example, the image P1K may be a noise image such as so-called white noise.

[0024] Next, details of an example of the learning process will be described. In FIG. 1, the information processing device 100 executes a learning process for learning a model M1 using an image group IG1, etc. (step S1). The information processing device 100 receives an image containing noise (noise image) as input, and learns the model M1 so as to output (predict) an image obtained by removing noise from the noise image by one step in the previous process. For example, the information processing device 100 learns the model M1 by receiving as input a noise image, text (character information) indicating the content of the image to be generated (e.g., an object or phenomenon to be included in the image), and information (stage information) indicating the noise level of the noise image. Objects, phenomena, etc. may be collectively referred to as "objects." Note that the stage information may not be included in the input of the model M1 when it is used as information (internal information) managed within the model M1.

[0025] The text input to the model M1 may be in any form as long as it indicates the content of the image to be generated. The text may be a character string (word) indicating an object. In this case, the text may be a word indicating a single object, such as a bear or a sofa. The text may also be a character string including words indicating each of multiple objects. In this case, the text may be a character string including words indicating each of multiple objects, such as a bear and a bird, a sofa and a woman, and particles such as parallel particles (juxtaposed particles). The text may also be a character string (sentence) that has meaning as a whole. In this case, the text may be a sentence indicating the content to be expressed in an image, such as a certain situation, such as a bear watching a flying bird or a woman in a dress sitting on a sofa. Note that the input to the model M1 described above is merely an example, and the model M1 may use any information as input.

[0026] 1, the information processing device 100 trains a model M1 so as to generate an image in which noise has been removed by one level from a noise image at one level. In Fig. 1, the information processing device 100 trains the model M1 using text (also referred to as "training text") indicating the content of an image P10 and level information (a value indicating the level to be processed).

[0027] For example, the information processing device 100 executes a learning process (process SR1 in FIG. 1) in which the model M1 is trained to output an image P1K-1 using an image P1K, a learning text (e.g., text TX1), and stage information as input. For example, the text TX1 is a sentence such as, "A woman wearing a dress is sitting on a sofa." The stage information may be, for example, a value corresponding to time t in FIG. 9. In this case, the stage information may be information such as noise removal stage "0" or noise stage "K." For example, the stage information may be information (0, 1, etc.) indicating the number of iterations of the process, for example, the number of iterations of the process. In this way, the information processing device 100 trains the model M1 to generate an image P1K-1, which is a noise image at stage #K-1 in the previous process, from the image P1K, which is a noise image at stage #K.

[0028] Furthermore, the information processing device 100 executes a learning process (process SR2 in FIG. 1) in which the model M1 is trained to receive as input an image P1K-1, which is a noise image of stage #K-1, a learning text, and stage information, and to output an image P1K-2, which is a noise image of stage #K-2. Similarly, the information processing device 100 executes a learning process (process SR3 in FIG. 1) in which the model M1 is trained to receive as input an image P1K-2, which is a noise image of stage #K-2, a learning text, and stage information, and to output an image P1K-3, which is a noise image of stage #K-3. Note that the processes between process SR3 and process SRK-2 in FIG. 1 are similar to the processes described above in that they receive as input a noise image and train to output an image that is one stage before the noise addition of the noise image, and therefore detailed description thereof will be omitted.

[0029] The information processing device 100 executes a learning process (process SRK-2 in FIG. 1) to train a model M1 so that image P13, which is a noise image of stage #3, training text, and stage information are input, and image P12, which is a noise image of stage #2. The information processing device 100 executes a learning process (process SRK-1 in FIG. 1) to train a model M1 so that image P12, which is a noise image of stage #2, training text, and stage information are input, and image P11, which is a noise image of stage #1.

[0030] Then, the information processing device 100 executes a learning process (process SRK in FIG. 1) to learn a model M1 so as to output image P10, which is an image at stage #0, i.e., an image without noise, using image P11, which is a noise image at stage #1, training text, and stage information as input. Processes SR1 to SRK in FIG. 1 correspond to the generation process (reverse process) in the diffusion model. In FIG. 1, the information processing device 100 learns a model M1 using an image group IG1 including images P10, P11, P12, ... P1K-2, P1K-1, P1K, etc.

[0031] In this way, the information processing device 100 learns the model M1, which is a model that generates data by inputting a noisy image and gradually removing noise. In Fig. 1, the information processing device 100 learns the model M1, which receives a noisy image, text, and stage information as inputs and gradually removes noise from the noisy image, thereby generating data (also referred to as a "generated image") that reflects the content indicated by the text.

[0032] 1, the information processing device 100 gradually removes noise from a noisy image to train a model M1 that generates a generated image showing the content of the text, for example, a situation (state) of a woman wearing a dress sitting on a sofa. In this way, the information processing device 100 trains a model M1 that converts a noisy image so as to approximate the training image based on the text.

[0033] 1 is a part of the learning of the model M1, and the information processing device 100 learns the model M1 using learning data including various image groups other than the image group IG1. The process shown in Fig. 1 is merely an example, and the information processing device 100 may learn the model M1 based on a learning method related to a diffusion model disclosed in Non-Patent Documents 2 and 3, etc.

[0034] In this way, the information processing device 100 learns a diffusion model that converts a noise image so as to approach the training image based on a training image including a predetermined object and text indicating the predetermined object. As a result, the information processing device 100 learns to gradually restore an image indicated by the text from the noise image, and learns model M1, which is a diffusion model trained to restore an image in accordance with attention based on the text.

[0035] In the above example, the information processing device 100 inputs an image into the diffusion model as text indicating the content of the image, and learns the diffusion model. For example, if text indicating the content of the image is also collected when the image is collected, the text is also known, and the information processing device 100 uses the data as learning data to learn a model such as model M1 through self-supervised learning. Note that if another device (learning device) learns a model such as a diffusion model, the information processing device 100 does not need to learn the diffusion model. If the information processing device 100 does not learn the diffusion model, the information processing device 100 may acquire model M1 by receiving the diffusion model, model M1, from the learning device.

[0036] [1-2. Example of generation process] Next, an example of an image generation process using model M1 will be described. Note that, for the sake of explanation, FIG. 1 illustrates an example in which the text used to generate the generated image (also referred to as "corresponding text") is text TX1, but the corresponding text is not limited to text TX1 and may be any image. For example, information processing device 100 receives a designation from the user that text TX1 is to be used to generate the generated image. In this way, the following describes an example in which text TX1 is text input by the user (also referred to as "input text").

[0037] The information processing device 100 executes a generation process to generate a generated image using a noise image, text, and stage information as inputs (step S11). For example, the noise image NZ1 may be the same noise image as the image P1K, or may be a different noise image. In FIG. 1, the information processing device 100 generates an image P20 by repeatedly generating images using the model M1, using the noise image NZ1, text TX1, and stage information as input information IN1 to the initial model M1.

[0038] For example, the information processing device 100 generates an image P20 by repeating input to the model M1 K times. For example, when K is 1000, the information processing device 100 repeats input to the model M1 999 times, using as input the noise image NZ1, the text TX1, and the image output by the model M1 to which the stage information has been input.

[0039] For example, in the first (first) iteration of the process, the information processing device 100 inputs a noise image NZ1, text TX1, and stage information indicating that this is the first (first) iteration of the process to the model M1, and causes the model M1 to output image #1. Then, in the second iteration of the process, the information processing device 100 inputs image #1, text TX1, and stage information indicating that this is the second iteration of the process to the model M1, and causes the model M1 to output image #2. In the third iteration of the process, the information processing device 100 inputs image #2, text TX1, and stage information indicating that this is the third iteration of the process to the model M1, and causes the model M1 to output image #3.

[0040] Similarly, in the 4th to K-1th iterations of the repetitive processing, the information processing device 100 inputs each of the images #3 to #K-2, the text TX1, and stage information corresponding to the processing to the model M1, and causes the model M1 to output each of the images #4 to #K-1. Then, in the Kth iteration of the repetitive processing, the information processing device 100 inputs the image #K-1, the text TX1, and stage information indicating that this is the Kth processing to the model M1, and causes the model M1 to output the image P20.

[0041] As a result, the information processing device 100 generates an image P20 showing the content of the text TX1 by gradually removing noise from the noise image. In FIG. 1, the information processing device 100 generates an image P20, which is a generated image expressing a situation in which a woman wearing a dress, included in the text TX1, is sitting on a sofa, by gradually removing noise from the noise image using the model M1. In this way, the information processing device 100 generates an image P20 showing the content of the text TX1 in the same way as image P10, by gradually removing noise from the noise image based on the text TX1.

[0042] [1-3. Example of information generation] The above-described processing makes it possible to generate an image using a diffusion model that generates an image showing the content of text from a noise image. Information generated during processing using the diffusion model can be used for purposes other than generating a generated image, and can also be used to generate other information. Therefore, the information processing device 100 generates other information, such as mask information, using information generated during processing using the diffusion model described above. First, before explaining an example of generating other information, an example of the configuration of model M1, which is a diffusion model, will be described using FIG. 9. FIG. 9 is a diagram showing an example of the configuration of the model.

[0043] The model structure example shown in FIG. 9 shows an example of the structure of model M1. In this case, model M1 has a structure based on U-net. Note that the structure based on U-net is merely an example, and model M1 is not limited to a structure based on U-net, and any structure can be adopted as long as the desired processing is possible. For example, model M1 may have a structure based on a technique other than U-net, such as Transformer. Note that since structures based on U-net are known, detailed explanations will be omitted, but a brief overview will be provided below.

[0044] In Figure 9, model M1, which has a structure based on U-net, has a structure that includes a plurality of first-type blocks, ResBlock, and second-type blocks, AttnBlock. For example, the first-type block, ResBlock, is a residual block that includes a combination of a convolutional layer and SkipConnection. The second-type block, AttnBlock, is a block that has an attention function, and its structure will be described later.

[0045] In the model shown in Figure 9, the left side corresponds to the input side, and the right side corresponds to the output side. The lines and arrows connecting each part indicate the concatenation relationship between each part. For example, model M1 applies a first type of block (ResBlock) and a second type of block (AttnBlock) to the input, and in the first half (the left side (input side) of Figure 9), it reduces the input resolution and increases the number of channels, and in the second half (the right side (output side) of Figure 9), it reduces the number of channels and increases the resolution.

[0046] The model structure shown in FIG. 9 has a structure in which pairs of first-type blocks and second-type blocks are connected in multiple layers. For example, "Z t The input portion corresponding to " corresponds to the portion where an image (noise image) corresponding to time t is input. For example, the input portion corresponding to "t" in FIG. 9 corresponds to the portion where information indicating the stage of the processing is input (stage information input portion). The stage information input portion is connected to each of the first type blocks and the second type blocks, and provides stage information to each block.

[0047] For example, "TextEncoder" in FIG. 9 corresponds to an encoding unit that executes a process of encoding input text. The encoding unit is connected to each of the second-type blocks and provides text information (encoding information) to each of the second-type blocks. Thus, in the model shown in FIG. 9, text information encoded by the encoding unit indicated by "TextEncoder" is input to each of the second-type blocks.

[0048] AttnBlock, which is a second type of block having an attention function, has a structure as shown in Fig. 10. Fig. 10 is a diagram showing an example of the configuration of the main part of the model. Specifically, Fig. 10 is a diagram showing an example of the configuration of the second type of block.

[0049] In the second type block shown in Fig. 10, the left side corresponds to the input side and the right side corresponds to the output side. As shown in Fig. 10, in the second type block (AttnBlock), information from the encoding unit is used in the part with cross-attention function (cross-attention unit) denoted as "Cross-Attn".

[0050] 10 includes, in addition to the cross-attention section, a section having a self-attention function denoted as "Self-Attn" and a section having a dropout function denoted as "Dropout." The configuration in the second-type block is not limited to that shown in FIG. 10, and any configuration can be adopted as long as it has a cross-attention section that uses information from text, and detailed description thereof will be omitted.

[0051] For example, the cross-attention unit calculates cross-attention between text information and image information. The cross-attention unit obtains a cross-attention result according to the content of the text. In this way, the cross-attention unit generates information (also called "attention information") indicating the cross-attention result between text information and image information.

[0052] An example of attention information generated at each stage will now be shown using Fig. 2. Fig. 2 is a diagram showing another example of information processing according to the embodiment. For the sake of explanation, Fig. 2 shows an example in which image P20 is generated from image P2K (corresponding to noise image NZ1 in Fig. 1), which is a noise image in Fig. 1.

[0053] The text TX1 is divided into a token TK1, which is designated as "token #1" in Fig. 2, and a token TX2, which is designated as "token #2" in Fig. 2. When the tokens TK1 and TK2 are not to be distinguished from each other, they are referred to as "token TK."

[0054] In Figure 2, token TK1 is "sofa" and token TX2 is "woman in a dress." For example, token TK1 and token TX2 correspond to each of the multiple elements that make up text TX1. For example, tokens TK1 and TX2 are character strings contained in the text, and are elements that indicate the content (meaning) of a certain object, phenomenon, etc. For example, a token is a unit for generating attention information in cross-attention.

[0055] 2, model M1 generates attention information corresponding to each token at each stage of processing. For example, model M1 generates attention information for the first processing in the process of generating image P2K-1 (image #1) in response to input of image P2K, text TX1, and stage information indicating the first (first) processing.

[0056] 2, for token TK1 (token #1), model M1 generates attention information A1K as attention information in the first processing. The attention information A1K is information indicating the area in the image (image P2K) that corresponds to token TK1.

[0057] Each piece of attention information, such as the attention information A1K shown in FIG. 2, is map information (attention map) that indicates, for example, the degree of relevance of each region (pixel, etc.) of an image to an object indicated by a token. In the example of FIG. 2, the darker the hatching in the attention information, the higher the relevance of the region (pixel, etc.) to the object indicated by the token in the corresponding image. For example, the darker the hatching in the attention information, the higher the possibility that the corresponding region contains the object indicated by the token. Also, the lighter the hatching in the attention information, the lower the possibility that the region (pixel, etc.) is related to the object indicated by the token in the corresponding image. For example, the lighter the hatching, the higher the possibility that the corresponding region does not contain the object indicated by the token. For example, the darker the hatching in the attention information, the higher the value of that portion, and the lighter the hatching, the lower the value of that portion.

[0058] 2, the model M1 generates attention information A2K for the token TK2 (token #2) as attention information in the first processing. The attention information A2K is information indicating the area corresponding to the token TK2 in the image (e.g., image P2K).

[0059] In addition, model M1 generates attention information for the second processing in the process of generating image P2K-2 (image #2) in response to input of image P2K-1 (image #1), text TX1, and stage information indicating that this is the second processing.

[0060] 2, the model M1 generates attention information A1K-1 for the token TK1 as attention information in the second processing. The attention information A1K-1 is information indicating the area corresponding to the token TK1 in an image (e.g., image P2K-1).

[0061] 2, the model M1 generates attention information A2K-1 for the token TK2 as attention information in the second processing. The attention information A2K-1 is information indicating the area corresponding to the token TK2 in an image (e.g., image P2K-1).

[0062] Similarly, in the 3rd to K-3rd iterations of the process, model M1 generates attention information for each of the 3rd to K-3rd iterations of the process to generate each of images #3 to #K-3 in response to input of each of images #2 to #K-4, text TX1, and stage information corresponding to that process.

[0063] In addition, model M1 generates attention information for the K-2nd processing in the process of generating image P22 (image #K-2) in response to input of image P23 (image #K-3), text TX1, and stage information indicating that this is the K-2nd processing.

[0064] In Fig. 2, model M1 generates attention information A13 for token TK1 as attention information in the (K-2)th processing. Attention information A13 is information indicating the area in an image (e.g., image P23) corresponding to token TK1. In Fig. 2, attention information A13 is assumed to be attention information that most appropriately indicates the area in the image where the object indicated by token TK1 is located.

[0065] 2, the model M1 generates attention information A23 for token TK2 as attention information for the (K-2)th processing. The attention information A23 is information indicating the area in an image (e.g., image P23) corresponding to token TK2. In FIG. 2, the attention information A23 is assumed to be attention information that most appropriately indicates the area in the image where the object indicated by token TK2 is located.

[0066] In addition, model M1 generates attention information for the K-1th processing in the process of generating image P21 (image #K-1) in response to input of image P22 (image #K-2), text TX1, and stage information indicating that this is the K-1th processing.

[0067] 2, the model M1 generates attention information A12 for the token TK1 as attention information in the (K−1)th processing. The attention information A12 is information indicating the area corresponding to the token TK1 in an image (e.g., image P22).

[0068] 2, the model M1 generates attention information A22 as attention information for the token TK2 in the (K−1)th processing. The attention information A22 is information indicating an area in an image (e.g., image P22) that corresponds to the token TK2.

[0069] In addition, model M1 generates attention information for the Kth processing in the process of generating image P20 (image #K) in response to input of image P21 (image #K-1), text TX1, and stage information indicating that it is the Kth processing.

[0070] 2, the model M1 generates attention information A11 as attention information for the Kth processing for the token TK1. The attention information A11 is information indicating an area in an image (for example, image P21) that corresponds to the token TK1.

[0071] 2, the model M1 generates attention information A21 for the token TK2 as attention information in the Kth processing. The attention information A21 is information indicating an area in an image (e.g., image P21) that corresponds to the token TK2.

[0072] In this way, in the model M1, attention information indicating the result of cross-attention between text information and image information is generated in the process of generating a generated image. Therefore, the information processing device 100 generates the following information using the attention information generated in the process of generating a generated image.

[0073] In the following, when describing attention information A1K, A1K-1, ... A12, A11, etc. corresponding to token TK1 (token #1) without distinction, it may be referred to as attention information A1, and when describing attention information A2K, A2K-1, ... A22, A21, etc. corresponding to token TK2 (token #2) without distinction, it may be referred to as attention information A2. Also, when describing attention information A1 and attention information A2, i.e., all of the attention information A1K, A2K, A1K-1, A2K-1, ... A12, A22, A11, A21, etc. in Figure 2 without distinction, it may be referred to as attention information A.

[0074] [1-3-1. Example of mask information generation] First, a first generation example in which the information processing device 100 generates mask information using attention information A will be described below with reference to Fig. 2. Note that explanations of points similar to those described above will be omitted as appropriate.

[0075] 2, the information processing device 100 generates an image P20 that is a corresponding image corresponding to the text TX1 using a model M1 and text TX1, and generates mask information of the target range indicated by the text TX1 in the image P20 based on attention information A used by the model M1 when generating the image P20. The information processing device 100 generates mask information of the target range indicated by the text TX1 in the image P20 based on attention information A at a predetermined stage during the generation process of the image P20 that is a generated image using the model M1.

[0076] For example, the information processing device 100 generates mask information for a target range indicated by a token TK in the image P20 based on attention information A corresponding to each token included in the text TX1 used by the model M1 when generating the image P20. The information processing device 100 acquires, from the user, designation information that designates one of the tokens TX1 and TX2 included in the text TX1, and generates mask information for a target range indicated by the one token in the image P20 based on attention information A corresponding to the one token indicated by the designation information.

[0077] The following describes an example in which the user specifies both tokens TX1 and TX2 included in text TX1. The information processing device 100 acquires, from the user, specification information specifying each of the multiple tokens TX1 and TX2 included in text TX1, and generates mask information for the target range indicated by each of the multiple tokens TX1 and TX2 in image P20 based on attention information A1 and A2 corresponding to each of the multiple tokens TX1 and TX2 indicated by the specification information.

[0078] For example, the information processing device 100 generates mask information for a target range indicated by the token TK1 in the image P20 based on attention information A1 corresponding to the token TK1 included in the text TX1 used by the model M1 when generating the image P20. For example, the information processing device 100 generates mask information for a target range indicated by the token TK1 in the image P20 based on attention information A1 at a stage determined based on information stored in the storage unit 120, etc. For example, the information processing device 100 generates mask information for a target range indicated by the token TK1 in the image P20 based on attention information A1 at a preset stage (designated stage).

[0079] Note that the attention information used when generating mask information is not limited to one piece of attention information, and multiple pieces of attention information may be used. In this case, the information processing device 100 may use attention information (synthesized attention information) calculated (generated) using any calculation method such as weighted averaging for multiple pieces of attention information corresponding to each of multiple processing times (timings). The following describes a case where one piece of attention information that most appropriately indicates the area where the target indicated by each token TK is located is selected and used.

[0080] 2, attention information A13 in the (K-2)th processing is attention information that most appropriately indicates the area in the image where the object indicated by token TK1 is located, and therefore information processing device 100 generates mask information for the object range indicated by token TK1 in image P20 based on attention information A13. For example, information processing device 100 generates mask information that distinguishes the range including the object "sofa" indicated by token TK1 (object range) from the other range (also referred to as "non-object range") in image P20 based on attention information A13.

[0081] For example, the information processing device 100 generates mask information that masks a non-target range of the image P20 that is a range outside the target range of the token TK1, based on the attention information A13. For example, the information processing device 100 generates, as mask information, a segmentation mask that indicates a segmentation of the object "sofa" indicated by the token TK1 in the image P20, based on the attention information A13. For example, the information processing device 100 generates, as mask information, a range of the attention information A13 whose value satisfies a predetermined condition, as mask information that indicates the segmentation of the object "sofa" indicated by the token TK1. For example, the information processing device 100 generates, as mask information, a range (pixels, etc.) of the attention information A13 whose value is equal to or greater than a predetermined value, as mask information that indicates the segmentation of the object "sofa" indicated by the token TK1. In FIG. 2, the information processing device 100 generates, as mask information, a hatched range of the attention information A13 that indicates the segmentation of the object "sofa" indicated by the token TK1. For example, the information processing apparatus 100 generates mask information that masks a range of the attention information A13 whose value is less than a predetermined value as a non-target range.

[0082] For example, the information processing device 100 generates mask information of a target range indicated by the token TK2 in the image P20 based on attention information A2 corresponding to the token TK2 included in the text TX1 used by the model M1 when generating the image P20. For example, the information processing device 100 generates mask information of a target range indicated by the token TK2 in the image P20 based on attention information A2 at a preset stage (designated stage).

[0083] 2, attention information A23 in the (K-2)th processing is attention information that most appropriately indicates the area in the image where the object indicated by token TK2 is located, and therefore information processing device 100 generates mask information for the object range indicated by token TK2 in image P20 based on attention information A23. For example, information processing device 100 generates mask information that distinguishes between the range (object range) including the object indicated by token TK2, "woman wearing a dress," and the non-object range in image P20 based on attention information A23.

[0084] For example, the information processing device 100 generates mask information based on the attention information A23 to mask a non-target range in image P20 that is a range outside the target range of token TK2. For example, the information processing device 100 generates, as mask information, a segmentation mask that indicates a segmentation of the object "woman wearing a dress" indicated by token TK2 in image P20 based on the attention information A23. For example, the information processing device 100 generates, as mask information, a range in attention information A23 whose values ​​satisfy a predetermined condition as mask information indicating the segmentation of the object "woman wearing a dress" indicated by token TK2. In FIG. 2, the information processing device 100 generates, as mask information indicating the hatched range in attention information A23 as the segmentation of the object "woman wearing a dress" indicated by token TK2. For example, the information processing device 100 generates mask information that masks, as a non-target range, a range in attention information A23 whose values ​​are less than a predetermined value.

[0085] As described above, the information processing device 100 can appropriately generate other information using information obtained when an image is generated using a model that generates an image. For example, the information processing device 100 uses text, a target image to be generated (also referred to as a "target image"), and an image obtained by gradually adding noise to the target image, and trains a diffusion model (such as model M1) in response to input of the noise image and text in each iteration of processing to generate an image that is one step closer to the target image from the input noise image. As a result, when the information processing device 100 receives input of an image and text, it can train the diffusion model (such as model M1) to gradually generate an image indicated by the text.

[0086] For example, the information processing device 100 receives designation of text and tokens from a user. The information processing device 100 generates an image corresponding to the text using a diffusion model (such as model M1). The information processing device 100 generates mask information from attention information corresponding to each token generated during image generation processing using the diffusion model (such as model M1). The information processing device 100 provides various services using the mask information and the generated image.

[0087] The information processing device 100 executes various information processing as described above. For example, the information processing device 100 generates mask information that extracts an image portion corresponding to text. For example, when a token is a noun, the information processing device 100 generates mask information that specifies a range (target range) that includes an object corresponding to the noun.

[0088] For example, when the token is a verb, the information processing device 100 generates mask information that specifies a range (target range) that includes an event corresponding to the verb. As a result, the information processing device 100 visualizes an event indicated by a verb such as "see" and generates mask information that specifies the range (target range). For example, the information processing device 100 can dynamically convey the range to a visually impaired person or the like by using embossed information or the like generated based on the mask information.

[0089] For example, when text specifying a state (behavioral state) such as an action or pose (posture, gesture, etc.) is used, the information processing device 100 generates mask information that specifies a range (target range) that includes an event corresponding to the behavioral state indicated by the text. As a result, in accordance with text such as "a person performing XX action," the information processing device 100 generates mask information that specifies a range (target range) that includes an event corresponding to "XX action" indicated by the text.

[0090] Furthermore, the information processing device 100 can generate mask information that combines multiple tokens. For example, in the case of "bear and bird," where the token "bear" and the token "bird" are connected by a parallel particle (logical disjunction), the information processing device 100 generates mask information (logical disjunction mask information) that identifies both a first target range that includes an object corresponding to the token "bear" and a second target range that includes an object corresponding to the token "bird." In this case, the information processing device 100 generates mask information that distinguishes the first target range and the second target range from other ranges. Note that the information processing device 100 may generate mask information that can distinguish both the first target range and the second target range by adding information that can distinguish between the first target range and the second target range.

[0091] Furthermore, when the tokens "flying in the sky" and "car" are in a logic (logical product) where one token describes another token, such as "a car flying in the sky," the information processing device 100 generates mask information (logical product mask information) that identifies a target range that includes an object corresponding to the token "flying in the sky," which is a combination of the tokens "flying in the sky" and "car." Note that the above is merely an example, and the information processing device 100 may generate desired information by any processing. In this way, the information processing device 100 can appropriately generate other information using information at the time of image generation using a model that generates an image.

[0092] Note that the information processing device 100 may generate mask information using any information. For example, the information processing device 100 may obtain cross-attention results (attention information) at multiple different resolutions depending on the positions of the second-type blocks using the cross-attention unit shown in Fig. 10. In this case, the information processing device 100 may select attention information according to the resolution and generate mask information from the selected attention information.

[0093] For example, the information processing device 100 may generate mask information for the target range indicated by the token TK1 in the image P20 based on attention information A1 corresponding to a resolution determined based on information stored in the storage unit 120, etc. For example, the information processing device 100 may generate mask information for the target range indicated by the token TK1 in the image P20 based on attention information A1 that is closest to a preset resolution (specified resolution). Furthermore, the information processing device 100 may also generate mask information for the target range indicated by the token TK2 in the image P20 based on attention information A2 corresponding to the resolution, as in the case of the token TK1.

[0094] Furthermore, information processing device 100 provides a service using the generated mask information. For example, information processing device 100 transmits the generated mask information to terminal device 10 used by a user. For example, information processing device 100 transmits to terminal device 10 used by the user information that associates mask information generated using information generated during the generation process of a generated image using model M1 with the generated image generated using model M1.

[0095] For example, the information processing device 100 transmits information in which a generated image generated by the model M1, mask information generated using information at the time of generation processing of the generated image, and a label indicating an object corresponding to the mask information, to the terminal device 10 used by the user. For example, the information processing device 100 transmits information in which a generated image generated by the model M1, mask information, and a label indicating an object corresponding to the mask information, as learning data, to the terminal device 10 used by the user.

[0096] Through the above-described processing, the information processing device 100 can appropriately generate any information, such as data used for learning (learning data). For example, the information processing device 100 can automatically generate a segmentation mask corresponding to any word in an input text. For example, one application is to use the generated image to improve accuracy in an image classification task. Therefore, by using the information generated by the information processing device 100 as learning data, it is possible to generate an image with a mask, which can be applied to classification tasks, detection tasks, segmentation tasks, etc.

[0097] The information processing system 1 (see FIG. 3) may also provide a UI (User Interface) for searching for an object corresponding to a portion of an image designated (touched) by a user. In this case, the information processing system 1 executes a process for searching for an object in a portion of an image designated by the user, thereby providing, for example, information indicating the object contained in the portion of the image touched by the user, for example, for young children. The information processing system 1 may also provide a service for assisting the creation of embossed books for visually impaired people, as described above, to help them understand the shapes of animals, etc. In addition, in the case of a book in which the embossing created by the information processing system 1 can be manipulated in real time, information (words) indicating the object corresponding to the portion of the image designated by the user can be provided. The information processing system 1 can also specify a word to identify an area in an image containing many objects. For example, the information processing system 1 can identify an area in an image containing a person performing a specific action.

[0098] The above-described process is merely an example, and the information processing device 100 may generate other information using various information. For example, the image used for learning the model M1 may be an image that satisfies a predetermined condition. A second generation example will be described in which the information processing device 100 generates information based on an image that satisfies a predetermined condition. Note that explanations of points similar to the above-described process will be omitted as appropriate.

[0099] For example, model M1 may be a diffusion model trained to use a target image that is an image satisfying a predetermined condition, text (corresponding text) associated with the target image, and a noise image to gradually restore the target image in accordance with attention based on the corresponding text. In this case, information processing device 100 may train model M1 to use a target image that is an image satisfying a predetermined condition, text associated with the target image, and a noise image to gradually restore the target image in accordance with attention based on the corresponding text.

[0100] For example, the image satisfying the predetermined condition may be a real image. For example, the image satisfying the predetermined condition may be a captured image (real image) of a predetermined object captured by an imaging device such as a camera. For example, the model M1 may be a diffusion model trained using a real image, corresponding text indicating the content of the real image, and a noise image to gradually restore the real image according to attention based on the corresponding text.

[0101] In this case, the information processing device 100 may train the model M1 using a real image, corresponding text indicating the content of the real image, and a noise image to gradually restore the real image according to attention based on the corresponding text. The information processing device 100 trains the model M1 using an inversion method. This enables the information processing device 100 to generate (reproduce) the real image using the model M1, which is a diffusion model (generative model).

[0102] In this case, the information processing device 100 generates mask information for a target range indicated by the text using a model M1 that generates (reproduces) a real image and the text. For example, the information processing device 100 generates a target image (hereinafter also referred to as "image P30") that is a real image corresponding to the text using the model M1, and generates mask information for a target range indicated by the text in the image P30 based on attention information used by the model M1 when generating the image P30.

[0103] In this way, the information processing device 100 generates mask information corresponding to a real image, which makes it easy to add real image data to training data for detection tasks and segmentation tasks. Therefore, the information processing device 100 can appropriately generate other information using information obtained when an image is generated using a model that generates an image. The information processing device 100 may train the model M1 that generates a real image using any training method. For example, the information processing device 100 may train the model M1 that generates a real image based on a training method related to a diffusion model disclosed in Non-Patent Document 4, etc.

[0104] As described above, the information processing device 100 receives an image and text representing the image, and trains a model to gradually restore the received image from a noise image and the text, restoring the image while taking into account attention based on the text. The information processing device 100 also generates mask information for the range indicated by the text based on the attention used by the model when generating the image from the text. For example, the information processing device 100 generates mask information by performing inversion learning. This allows the information processing device 100 to generate mask information corresponding to specified character information.

[0105] Furthermore, the information processing device 100 may perform various processes other than just generating mask information. For example, the information processing device 100 performs processing on a generated image (target image) based on the mask information. The information processing device 100 may generate a processed image to be used as learning data for machine learning by processing the target image. For example, the information processing device 100 generates a processed image in which a range of the target image other than the target range identified based on the mask information is used as the background.

[0106] For example, the information processing device 100 generates mask information that distinguishes the range (target range) of the target image (image P30) that includes the object indicated by the text TX2 from the other range (non-target range) based on attention information generated during the target image generation process using model M1 with text TX2 as input.

[0107] For example, the information processing device 100 identifies the non-target area indicated by the generated mask information as the background of the target image (image P30). Then, the information processing device 100 generates a processed image by performing a predetermined processing process using the non-target area as the background of the target image (image P30). For example, the information processing device 100 generates an image P31 as the processed image by removing the non-target area as the background of the target image (image P30).

[0108] In this way, the information processing device 100 can also be applied to tasks such as removing the background of a real image. The information processing device 100 may dynamically change (determine) which part of an image is to be used as the background depending on the category in which the image is used (for example, the service field).

[0109] This point will be described using the image P20 shown in FIG. 2 as an example. For example, when the category is the interior category, the information processing device 100 specifies the object "sofa" indicated by the token TK1 in the image P20 as the object range, and the rest as the non-object range. Then, the information processing device 100 removes the range (non-object range) other than the object range of the object "sofa" indicated by the token TK1 in the image P20 as the background. As a result, the information processing device 100 generates, as a processed image, an image P32 in which the range (non-object range) other than the object range of the object "sofa" indicated by the token TK1 in the image P20 has been removed as the background.

[0110] Furthermore, for example, when the category is fashion, the information processing device 100 specifies the portion of the image P20 corresponding to the object "woman wearing a dress" indicated by the token TK2 as the object range, and the rest as the non-object range. Then, the information processing device 100 removes the range (non-object range) other than the object range of the object "woman wearing a dress" indicated by the token TK2 from the image P20 as the background. As a result, the information processing device 100 generates, as a processed image, an image P33 from which the range (non-object range) other than the object range of the object "woman wearing a dress" indicated by the token TK2 from the image P20 has been removed as the background.

[0111] [1-3-2. Examples of generating other information] Note that the information processing device 100 may generate various kinds of feature information such as feature vectors, not limited to mask information. For example, the information processing device 100 generates a corresponding image corresponding to the input text using a diffusion model and the input text, and generates feature information indicating the features of the target range indicated by the input text.

[0112] In this case, the information processing device 100 may use the diffusion model and the input text to generate a corresponding image corresponding to the input text, and generate feature information indicating the features of the target range indicated by the input text within the corresponding image based on the attention used by the diffusion model when generating the corresponding image. A third generation example in which the information processing device 100 generates feature information other than mask information using attention information A will be described below. Note that explanations of points similar to those described above will be omitted as appropriate.

[0113] 2, the information processing device 100 generates an image P20 that is a corresponding image corresponding to the text TX1 using the model M1 and the text TX1, and generates feature information indicating the features of the target range indicated by the text TX1 in the image P20 based on the attention information A used by the model M1 when generating the image P20. For example, the information processing device 100 generates, as the feature information, a feature vector obtained by extracting the features of the target indicated by the text TX1 included in the image P20.

[0114] For example, the information processing device 100 generates feature information to be used as learning data for machine learning. For example, the information processing device 100 may generate a feature vector indicating the features of an object indicated by text TX1 included in image P20 as supervised answer information for a feature extraction model that extracts image features. For example, the information processing device 100 generates a feature vector indicating the features of an object indicated by text TX1 as supervised answer information to be output by the feature extraction model when an image including the object indicated by text TX1 is input.

[0115] For example, the information processing device 100 generates a feature vector indicating the features of an object indicated by text using model M2, which is a feature extraction model that extracts features of an input image. Model M2, which is a feature extraction model, is a model that outputs feature information of an object indicated by text in an image. Note that the feature extraction model that extracts the features of an image may be any model as long as it is capable of extracting features from an image, and detailed description thereof will be omitted. For example, when model M2 is learned by a device (learning device) other than the information processing device 100, the information processing device 100 may acquire model M2 by receiving model M2 from the learning device. Note that the information processing device 100 may learn model M2.

[0116] For example, the information processing device 100 may generate a feature vector indicating the characteristics of the target indicated by the text in the extraction target image by using, as the extraction target image, a processed image generated using the attention information A. For example, the information processing device 100 may generate a feature vector indicating the characteristics of the target indicated by the text in the extraction target image by using, as the extraction target image, a processed image that has been subjected to background removal processing using the attention information A.

[0117] The information processing device 100 generates feature information of the object indicated by the text in the extraction target image using the extraction target image based on attention information A and model M2, which is a feature extraction model. The information processing device 100 inputs the extraction target image based on attention information A to model M2, and causes model M2 to output a feature vector indicating the feature of the object indicated by the text in the extraction target image. In this way, the information processing device 100 generates a feature vector indicating the feature of the object indicated by the text in the extraction target image.

[0118] The information processing device 100 generates a feature vector indicating the characteristics of the target indicated by the text TX2 in the image P31 by inputting the image P31, which has been generated using the attention information A corresponding to the text TX2 and from which the background has been removed, into the model M2 as an image to be extracted. In this way, the information processing device 100 generates a feature vector indicating the characteristics of the target range indicated by the text TX2, "A woman in a dress is sitting on a sofa."

[0119] The information processing device 100 generates a feature vector indicating the characteristics of the range (target range) that includes the object "sofa" indicated by the token TK1 in the image P32 by inputting the image P32, which has been generated using the attention information A1 corresponding to the token TK1 and from which the background has been removed, into the model M2 as an extraction target image. In this way, the information processing device 100 generates a feature vector indicating the characteristics of the target range that corresponds to the object "sofa" indicated by the token TK1.

[0120] The information processing device 100 generates a feature vector indicating the characteristics of the range (target range) that includes the object "woman wearing a dress" indicated by the token TK2 in the image P33 by inputting the image P33, which has been generated using the attention information A2 corresponding to the token TK2 and from which the background has been removed, into the model M2 as an image to be extracted. In this way, the information processing device 100 generates a feature vector indicating the characteristics of the target range that corresponds to the object "woman wearing a dress" indicated by the token TK2.

[0121] In this way, the information processing device 100 can appropriately generate other information using information obtained when an image is generated using a model that generates an image. As described above, the information processing device 100 generates a corresponding image indicated by text using a diffusion model, and generates mask information for a target range indicated by the text based on attention information used by the diffusion model when generating the corresponding image. Then, the information processing device 100 applies a mask corresponding to the text to the corresponding image, thereby generating training data for learning features of the target image indicated by the text. For example, the information processing device 100 generates a feature vector as feature information, as in the example described above.

[0122] The above-described process is merely an example, and the information processing device 100 may generate various feature information by appropriately using various information. For example, the information processing device 100 may generate information by processing using mask information instead of global average pooling processing in the model. In this case, the information processing device 100 may generate a feature vector by using the obtained mask information and performing pooling that emphasizes that part. For example, when the category is the interior category, the information processing device 100 generates a feature vector by using mask information corresponding to the object "sofa" indicated by token TK1 in image P20. For example, when the category is the fashion category, the information processing device 100 generates a feature vector by using mask information corresponding to the object "woman wearing a dress" indicated by token TK2 in image P20.

[0123] Furthermore, information processing device 100 provides a service using the generated feature information. For example, information processing device 100 transmits the generated feature information to terminal device 10 used by a user. For example, information processing device 100 transmits information that associates feature information generated using information generated during the generation process of a generated image in model M1 with the generated image generated in model M1 to terminal device 10 used by the user.

[0124] For example, the information processing device 100 transmits information in which a generated image generated by the model M1, feature information generated using information at the time of the generation process of the generated image, and a label indicating an object corresponding to the feature information, to the terminal device 10 used by the user. For example, the information processing device 100 transmits information in which a generated image generated by the model M1, feature information, and a label indicating an object corresponding to the feature information, as learning data, to the terminal device 10 used by the user.

[0125] [2. Information Processing System Configuration] Next, a configuration example of the information processing system 1 will be described with reference to Fig. 3. Fig. 3 is a diagram showing a configuration example of the information processing system according to the embodiment. As shown in Fig. 3, the information processing system 1 includes an information processing device 100, an information providing device 50, and a terminal device 10. The information processing device 100, the information providing device 50, and the terminal device 10 are connected to each other via a predetermined network N so as to be able to communicate with each other via wired or wireless communication. Note that the information processing system 1 shown in Fig. 3 may include a plurality of information processing devices 100, a plurality of information providing devices 50, and a plurality of terminal devices 10.

[0126] The information processing device 100 is a computer that generates information using a model such as a diffusion model. That is, the information processing device 100 is a generation device that performs information generation processing. The information processing device 100 generates a corresponding image corresponding to the text using the diffusion model and text, and generates mask information for a target range indicated by the text in the corresponding image based on the attention used by the diffusion model when generating the corresponding image.

[0127] Furthermore, information processing device 100 uses a diffusion model and corresponding text, which is text associated with a generated image that is an image that satisfies predetermined conditions, to generate mask information for a target range indicated by the corresponding text in the generated image based on the attention used by the diffusion model when generating the generated image. Information processing device 100 uses the diffusion model and the text to generate a corresponding image that corresponds to the text, and generates feature information that indicates the features of the target range indicated by the text.

[0128] The information processing device 100 is a computer that provides a service to a user who uses a terminal device 10. The information processing device 100 provides a service using generated mask information. The information processing device 100 provides the generated mask information to the user. The information processing device 100 transmits the generated mask information to the terminal device used by the user.

[0129] The information processing device 100 provides a service using the generated characteristic information. The information processing device 100 provides the generated characteristic information to a user. The information processing device 100 transmits the generated characteristic information to a terminal device 10 used by the user.

[0130] The information providing device 50 is a server device that provides the information processing device 100 with information (data) that can be used for model learning. For example, the information providing device 50 transmits data that is permitted to be used for model learning to the information processing device 100 as learning data. For example, the information providing device 50 publishes public data on the Internet, and in response to a request from an external device such as the information processing device 100, transmits the requested data to the external device that is the request destination. For example, the information providing device 50 receives request information requesting data from the information processing device 100, and in response to the received request information, transmits data corresponding to the request information to the information processing device 100.

[0131] For example, the information providing device 50 transmits, as learning data to the information processing device 100, information associating a group of images in which noise has been added in stages to a single image with text indicating an object included in the single image. For example, the information providing device 50 transmits, as learning data to the information processing device 100, data associating a plurality of staged images in which noise has been added in stages to images of a transaction object to be traded on an e-commerce site or the like with text indicating the content of the transaction object.

[0132] The terminal device 10 is an information processing device used by a user. The terminal device 10 is realized by, for example, a smartphone, a tablet terminal, a notebook PC (Personal Computer), a desktop PC, a mobile phone, a PDA (Personal Digital Assistant), etc. The terminal device 10 may be referred to as a user. In other words, the user can be read as the terminal device 10.

[0133] The terminal device 10 accepts operations by a user. In addition, the terminal device 10 transmits various information to the information processing device 100 and the information providing device 50 in response to the user's operations. For example, the terminal device 10 transmits request information requesting various information from the information processing device 100 and the information providing device 50. In addition, the terminal device 10 receives various information from the information processing device 100 and the information providing device 50.

[0134] The terminal device 10 transmits an image (target image) designated by the user as a target for generating information to the information processing device 100. The terminal device 10 transmits information indicating the target designated by the user (designated information) to the information processing device 100.

[0135] The terminal device 10 receives various types of information from the information processing device 100. For example, the terminal device 10 receives mask information from the information processing device 100. For example, the terminal device 10 receives feature information from the information processing device 100.

[0136] The terminal device 10 receives, as learning data, information in which a generated image generated by the model M1, mask information, and a label indicating an object corresponding to the mask information are associated with each other from the information processing device 100. The terminal device 10 receives, as learning data, information in which a generated image generated by the model M1, feature information, and a label indicating an object corresponding to the feature information are associated with each other from the information processing device 100.

[0137] 3. Configuration of Information Processing Device Next, the configuration of the information processing device 100 according to the embodiment will be described with reference to Fig. 4. Fig. 4 is a diagram showing an example of the configuration of the information processing device 100 according to the embodiment. As shown in Fig. 4, the information processing device 100 has a communication unit 110, a storage unit 120, and a control unit 130. Note that the information processing device 100 may also have an input unit (e.g., a keyboard, a mouse, etc.) that accepts various operations from an administrator of the information processing device 100, and a display unit (e.g., a liquid crystal display, etc.) that displays various information.

[0138] (Communication unit 110) The communication unit 110 is realized by, for example, a network interface card (NIC), etc. The communication unit 110 is connected to a network via a wired or wireless connection, and transmits and receives information to and from the terminal device 10 and the information providing device 50.

[0139] (Storage unit 120) The storage unit 120 is realized by, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk. As shown in FIG. 4 , the storage unit 120 according to the embodiment includes a learning data storage unit 121 and a model information storage unit 122.

[0140] (Learning data storage unit 121) The learning data storage unit 121 according to the embodiment stores various information (learning data) related to data used for learning. The learning data storage unit 121 stores teacher data used for generating a model. The learning data storage unit 121 stores learning data information including information such as data used for input and correct answer information (correct answer image) corresponding to the data.

[0141] The learning data storage unit 121 stores learning data in which a combination of a noise image and text used for input is associated with image data (correct image) corresponding to the input.

[0142] The learning data storage unit 121 may store various types of information depending on the purpose, not limited to the above. For example, the learning data storage unit 121 stores information indicating the type of input data in association with each piece of input data.

[0143] (Model information storage unit 122) The model information storage unit 122 according to the embodiment stores information about a model. For example, the model information storage unit 122 stores information (model data) about a trained model (model) trained (generated) by a training process. FIG. 5 is a diagram illustrating an example of the model information storage unit according to the embodiment. In the example shown in FIG. 5, the model information storage unit 122 includes items such as "model ID," "purpose," and "model data."

[0144] "Model ID" indicates identification information for identifying a model. "Use" indicates the use of the corresponding model. "Model Data" indicates the data of the model. Figure 5 shows an example in which conceptual information such as "MDT1" is stored in "Model Data," but in reality, various information that constitutes the model, such as information on the model configuration (network configuration) and information on parameters, is included. For example, "Model Data" includes information including the nodes in each layer of the network, the functions used by each node, the connection relationships between the nodes, and the connection coefficients set for the connections between the nodes.

[0145] In FIG. 5, the model (model M1) identified by the model ID "M1" has an application of "image generation," and for example, model M1 indicates that it is a model used to generate an image showing the content of text. For example, model M1 indicates that it is a model used to convert character information into image information, that is, to express (reproduce) the content of the character information as an image. Also, it indicates that the model data of model M1 is model data MDT1.

[0146] The model information storage unit 122 may store various types of information depending on the purpose, not limited to the above. Models such as model M1 may be intended to be used as program modules that are part of artificial intelligence software. Models such as model M1 may also be programs.

[0147] (control unit 130) 4, the control unit 130 is a controller, and is realized by, for example, a CPU (Central Processing Unit) or an MPU (Micro Processing Unit) executing various programs (corresponding to examples of information processing programs such as learning programs and generation programs) stored in a storage device inside the information processing device 100 using a RAM as a work area. The control unit 130 is also a controller, and is realized by, for example, an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0148] 4, the control unit 130 has an acquisition unit 131, a generation unit 132, a learning unit 133, a processing unit 134, and a provision unit 135, and realizes or executes the functions and actions of information processing described below. Note that the internal configuration of the control unit 130 is not limited to the configuration shown in FIG. 4, and may be any other configuration as long as it performs the information processing described below.

[0149] (Acquisition part 131) The acquisition unit 131 acquires various types of information. The acquisition unit 131 acquires various types of information from an external information processing device such as the terminal device 10 or the information providing device 50. The acquisition unit 131 may receive various types of information indicated in the learning data storage unit 121, the model information storage unit 122, etc. from the external information processing device.

[0150] The acquisition unit 131 acquires various pieces of information from the storage unit 120. The acquisition unit 131 acquires various pieces of information from the learning data storage unit 121, the model information storage unit 122, etc. The acquisition unit 131 receives various pieces of information from an external device.

[0151] The acquisition unit 131 receives various information from the terminal device 10 used by the user. For example, the acquisition unit 131 receives an image for which mask information is to be generated from the terminal device 10. For example, the acquisition unit 131 receives information from the terminal device 10 that specifies an object in the image for which mask information is to be generated. For example, the acquisition unit 131 receives an image for which feature information is to be generated from the terminal device 10.

[0152] The acquisition unit 131 receives various types of information from the information providing device 50. For example, the acquisition unit 131 receives information used for learning from the information providing device 50. For example, the acquisition unit 131 receives learning data used for learning the model M1 from the information providing device 50.

[0153] The acquisition unit 131 acquires a diffusion model, which is a model trained to gradually restore an image indicated by text from a noise image and to restore an image in accordance with attention based on the text. The acquisition unit 131 acquires input text input by a user. The acquisition unit 131 acquires designation information that designates one token from among the tokens included in the input text. The acquisition unit 131 acquires designation information that designates multiple tokens from among the tokens included in the input text.

[0154] The acquisition unit 131 acquires a target image, which is an image that satisfies a predetermined condition, and a corresponding text, which is text associated with the target image. The acquisition unit 131 acquires a diffusion model, which is a model that has been trained to gradually restore the target image from a noise image and the corresponding text, and which restores the target image in accordance with attention based on the corresponding text.

[0155] The acquisition unit 131 acquires a target image which is an actual image and corresponding text associated with the target image. The acquisition unit 131 acquires a target image which is a captured image of a predetermined target and corresponding text indicating the predetermined target. The acquisition unit 131 acquires corresponding text indicating the captured target captured by the target image. The acquisition unit 131 acquires corresponding text indicating the captured target included in the target image.

[0156] The acquisition unit 131 acquires a target image to be processed from a user, and acquires designation information from the user that designates a target in the image for which information is to be generated.

[0157] (Generation unit 132) The generation unit 132 executes a generation process to estimate various pieces of information. For example, the generation unit 132 executes the generation process based on various pieces of information acquired by the acquisition unit 131. The generation unit 132 executes the generation process based on various pieces of information stored in the storage unit 120. The generation unit 132 executes the generation process based on various pieces of information received from an external information processing device. The generation unit 132 executes the generation process based on a model learned by the learning unit 133. The generation unit 132 executes the generation process based on information determined by the processing unit 134.

[0158] The generation unit 132 generates a corresponding image corresponding to the input text using the diffusion model and input text acquired by the acquisition unit 131, and generates mask information of a target range indicated by the input text in the corresponding image based on the attention used by the diffusion model when generating the corresponding image. The generation unit 132 generates a corresponding image corresponding to the input text, and generates mask information of a target range indicated by a token in the corresponding image based on the attention corresponding to each token included in the input text used by the diffusion model when generating the corresponding image.

[0159] The generation unit 132 generates mask information for a target range indicated by one token in the corresponding image based on attention corresponding to one token specified by the designation information among the tokens included in the input text. The generation unit 132 generates mask information for a target range indicated by multiple tokens in the corresponding image based on attention corresponding to each of multiple tokens specified by the designation information among the tokens included in the input text.

[0160] The generating unit 132 generates mask information for a target range corresponding to the logical sum of a plurality of tokens.The generating unit 132 generates mask information for a target range corresponding to the logical product of a plurality of tokens.

[0161] The generation unit 132 generates mask information for a target range indicated by a token in a corresponding image based on attention corresponding to each token that is a noun included in the input text. The generation unit 132 generates mask information for a target range indicated by a token in a corresponding image based on attention corresponding to each token that is a verb included in the input text. The generation unit 132 generates a corresponding image that corresponds to the input text, and generates mask information for a target range indicated by the input text in a corresponding image based on attention at a predetermined stage during the diffusion model generation process.

[0162] The generation unit 132 generates mask information for a target range indicated by the corresponding text in the target image based on the attention used by the diffusion model when generating the target image, using the diffusion model and corresponding text acquired by the acquisition unit 131. The generation unit 132 generates mask information for a target range corresponding to a predetermined target in the target image based on the attention used by the diffusion model when generating the target image.

[0163] The generation unit 132 generates mask information for a target range of the target image corresponding to the captured target, based on the attention used by the diffusion model when generating the target image. The generation unit 132 generates a corresponding image corresponding to the corresponding text, and generates mask information for a target range of the corresponding image indicated by the corresponding text, based on the attention used at a predetermined stage during the generation process of the diffusion model.

[0164] The generation unit 132 generates a corresponding image corresponding to the input text using the diffusion model acquired by the acquisition unit 131 and the input text, and generates feature information indicating the features of the target range indicated by the input text. The generation unit 132 generates a vector indicating the features of the target range indicated by the input text as the feature information. The generation unit 132 generates the feature information to be used as learning data for machine learning.

[0165] The generation unit 132 generates a corresponding image corresponding to the input text, and generates feature information indicating the features of the target range indicated by the input text within the corresponding image based on the attention used by the diffusion model when generating the corresponding image. The generation unit 132 generates a corresponding image corresponding to the input text, and generates feature information indicating the features of the target range indicated by the token within the corresponding image based on the attention corresponding to each of the tokens included in the input text used by the diffusion model when generating the corresponding image.

[0166] The generation unit 132 generates feature information indicating the features of the target range indicated by one token in the corresponding image based on attention corresponding to one token specified by the designation information among the tokens included in the input text. The generation unit 132 generates feature information indicating the features of the target range indicated by multiple tokens in the corresponding image based on attention corresponding to each of multiple tokens specified by the designation information among the tokens included in the input text.

[0167] The generating unit 132 generates feature information indicating the feature of a target range corresponding to the logical sum of a plurality of tokens.The generating unit 132 generates feature information indicating the feature of a target range corresponding to the logical product of a plurality of tokens.

[0168] The generation unit 132 generates feature information indicating the characteristics of the target range indicated by the tokens of the corresponding images based on attention corresponding to each token, which is a noun included in the input text. The generation unit 132 generates feature information indicating the characteristics of the target range indicated by the tokens of the corresponding images based on attention corresponding to each token, which is a verb included in the input text. The generation unit 132 generates corresponding images corresponding to the input text, and generates feature information indicating the characteristics of the target range indicated by the input text of the corresponding images based on attention at a predetermined stage during the diffusion model generation process.

[0169] (Learning Section 133) The learning unit 133 learns various types of information. The learning unit 133 generates various models through a learning process. For example, the learning unit 133 performs the learning process based on information stored in the learning data storage unit 121, etc. The learning unit 133 generates various models based on information stored in the learning data storage unit 121, etc. For example, the learning unit 133 learns a model M1 using the learning data stored in the learning data storage unit 121.

[0170] The learning unit 133 learns a diffusion model, which is a model trained to gradually restore an image indicated by text from a noise image and to restore an image in response to attention based on the text. The learning unit 133 learns a diffusion model, which is a model trained to gradually restore a target image from a noise image and corresponding text and to restore the target image in response to attention based on the corresponding text.

[0171] The learning unit 133 learns a diffusion model using the plurality of stage images acquired by the acquisition unit 131. The learning unit 133 learns the diffusion model so as to generate a stage image that is an image in which noise has been removed by one stage from one of the plurality of stage images. The learning unit 133 learns the diffusion model so as to generate a first-stage image in which noise has been removed by one stage from a second-stage image. The learning unit 133 learns the diffusion model so as to generate a second-stage image in which noise has been removed by one stage from a third-stage image.

[0172] For example, the learning unit 133 performs a learning process based on various information acquired by the acquisition unit 131. For example, the learning unit 133 learns a model based on information generated by the generation unit 132. The learning unit 133 performs a learning process based on information from an external information processing device or information stored in the storage unit 120. The learning unit 133 performs a learning process based on information stored in the model information storage unit 122. The learning unit 133 stores the model generated by learning in the model information storage unit 122.

[0173] The learning unit 133 performs a learning process. The learning unit 133 performs various types of learning. The learning unit 133 learns various types of information based on the information acquired by the acquisition unit 131. The learning unit 133 learns (generates) a model. The learning unit 133 learns various types of information such as a model. The learning unit 133 generates a model through learning. The learning unit 133 learns the model using various machine learning techniques. For example, the learning unit 133 learns parameters of the model (network). The learning unit 133 learns the model using various machine learning techniques.

[0174] The learning unit 133 generates various learning models such as the model M1. The learning unit 133 learns network parameters. For example, the learning unit 133 learns network parameters of various learning models such as the model M1. The learning unit 133 performs a learning process using learning data stored in the model information storage unit 122, thereby generating various learning models such as the model M1. For example, the learning unit 133 generates a model used for generating an image. The learning unit 133 generates various learning models such as the model M1 by learning network parameters of various learning models such as the model M1.

[0175] The learning unit 133 performs a learning process based on the learning data (teacher data) stored in the model information storage unit 122. The learning unit 133 performs a learning process using the learning data stored in the model information storage unit 122, thereby generating various learning models such as model M1.

[0176] For example, the learning unit 133 performs a learning process using a method such as backpropagation (backpropagation method) so that an image output by the model M1 approaches correct answer information (information indicating a desired generation result) associated with input information input to the model M1. For example, the learning unit 133 performs a learning process so that an image output by the model M1 to which input information including a noise image has been input approaches an image (correct answer information) in which noise has been removed from the input noise image by one step.

[0177] For example, the learning unit 133 adjusts the values ​​of weights (i.e., connection coefficients) that are taken into account when values ​​are transmitted between nodes through the learning process. In this way, the learning unit 133 learns the model M1 through processing such as backpropagation, which corrects parameters (connection coefficients) so as to reduce the error between the output of the model M1 and the correct information corresponding to the input. For example, the learning unit 133 generates the model M1 by performing processing such as backpropagation so as to minimize a predetermined loss function. This allows the learning unit 133 to perform a learning process to learn the parameters of the model M1.

[0178] The model learning method is not limited to the above-described method, and any known technique can be applied. Each model may be generated using various conventional machine learning techniques, as appropriate. For example, the model may be generated using a supervised machine learning technique such as SVM (Support Vector Machine). Furthermore, the model may be generated using an unsupervised machine learning technique. For example, the model may be generated using a deep learning technique. For example, the model may be generated using various deep learning techniques, such as DNN (Deep Neural Network), RNN (Recurrent Neural Network), or CNN (Convolutional Neural Network). The above description of the model generation is merely an example, and the model may be generated using a learning technique appropriately selected depending on the obtainable information, etc. In other words, the learning unit 133 may generate the model M1 using any technique as long as the model M1 can be trained to output information corresponding to the correct answer information when input information included in the learning data is input.

[0179] As described above, the learning method used by the learning unit 133 is not particularly limited. For example, learning data in which data (input information) and its correct answer information (output information) are linked may be prepared, and the learning data may be input to a computational model based on a multilayer neural network for learning. Alternatively, a method based on DNNs such as CNN and 3D-CNN may be used. Specifically, the learning method used by the learning unit 133 may be a method based on a U-net. When targeting time-series data such as speech, the learning unit 133 may use a method based on a recurrent neural network (RNN) or LSTM (Long Short-Term Memory units), which is an extended version of an RNN.

[0180] (Processing unit 134) The processing unit 134 processes various types of information. The processing unit 134 performs information processing based on the information stored in the storage unit 120. The processing unit 134 functions as an inference unit that performs inference processing. The processing unit 134 performs inference processing using the model stored in the model information storage unit 122. For example, the learning unit 133 performs inference processing based on the information generated by the generation unit 132.

[0181] The processing unit 134 performs processing on the target image based on the mask information. The processing unit 134 generates a processed image to be used as learning data for machine learning by processing the target image. The processing unit 134 generates a processed image in which the range of the target image other than the target range specified based on the mask information is used as the background.

[0182] The processing unit 134 performs inference using the model M1 learned by the learning unit 133. The processing unit 134 performs inference using the model M1, which is a diffusion model. The processing unit 134 inputs a noise image and text as input information to the model M1, and performs inference processing to generate an image including an object corresponding to the text.

[0183] (Provider 135) The providing unit 135 provides various information. For example, the providing unit 135 provides various information to the terminal device 10 and the information providing device 50. The providing unit 135 transmits various information to the terminal device 10 and the information providing device 50. The providing unit 135 provides various information based on the various information acquired by the acquiring unit 131. The providing unit 135 provides various information based on the various information generated by the generating unit 132.

[0184] The providing unit 135 provides various pieces of information based on the various pieces of information learned by the learning unit 133. The providing unit 135 provides various pieces of information based on the model learned by the learning unit 133. The providing unit 135 provides various pieces of information based on the various pieces of information processed by the processing unit 134. For example, the providing unit 135 transmits the information to the terminal device 10.

[0185] The providing unit 135 provides a service using the diffusion model learned by the learning unit 133. The providing unit 135 provides a service using the mask information generated by the generating unit 132. The providing unit 135 provides the mask information generated by the generating unit 132 to a user. The providing unit 135 transmits the mask information generated by the generating unit 132 to a terminal device used by the user.

[0186] The providing unit 135 provides a service using the feature information generated by the generating unit 132. The providing unit 135 provides the user with the feature information generated by the generating unit 132. The providing unit 135 transmits the feature information generated by the generating unit 132 to the terminal device 10 used by the user.

[0187] [4. Information processing flow] Next, the flow of information processing by the information processing device 100 will be described with reference to Fig. 6 to Fig. 8. Fig. 6 to Fig. 8 are flowcharts showing an example of information processing according to the embodiment.

[0188] First, a processing example shown in Fig. 6 will be described. As shown in Fig. 6, the information processing device 100 acquires a diffusion model, which is a model trained to gradually restore an image indicated by text from a noise image and to restore an image in accordance with attention based on the text, and an input text input by a user (step S101).

[0189] The information processing device 100 uses the diffusion model and the input text to generate a corresponding image corresponding to the input text, and generates mask information that masks the target range indicated by the input text in the corresponding image based on the attention used when the diffusion model generated the corresponding image (step S102).

[0190] Next, a processing example shown in Fig. 7 will be described. As shown in Fig. 7, the information processing device 100 acquires a target image that is an image that satisfies a predetermined condition, a corresponding text that is text associated with the target image, and a diffusion model that is a model that has been trained to gradually restore the target image from a noise image and the corresponding text, and that restores the target image in accordance with attention based on the corresponding text (step S201).

[0191] The information processing device 100 uses the diffusion model and the corresponding text to generate mask information that masks the target range indicated by the corresponding text in the target image based on the attention used when the diffusion model generated the target image (step S202).

[0192] Next, a processing example shown in Fig. 8 will be described. As shown in Fig. 8, the information processing device 100 acquires a diffusion model, which is a model trained to gradually restore an image indicated by text from a noise image and to restore an image in accordance with attention based on the text, and an input text input by a user (step S301).

[0193] The information processing device 100 generates a corresponding image corresponding to the input text using the diffusion model and the input text, and generates feature information indicating the features of the target range indicated by the input text (step S302).

[0194] [5. Effects] As described above, the information processing device 100 according to the embodiment includes an acquisition unit 131 and a generation unit 132. The acquisition unit 131 acquires a diffusion model, which is a model trained to gradually restore an image indicated by text from a noise image and to restore the image in accordance with attention based on the text, and input text input by a user. The generation unit 132 generates a corresponding image corresponding to the input text using the diffusion model acquired by the acquisition unit 131 and the input text, and generates mask information for a target range indicated by the input text within the corresponding image based on the attention used by the diffusion model when generating the corresponding image.

[0195] In this way, the information processing device 100 according to the embodiment generates a corresponding image corresponding to the input text using the input text and a diffusion model trained to restore an image according to attention based on the text, and generates mask information for the target range of the corresponding image indicated by the input text based on the attention used by the diffusion model when generating the corresponding image. Therefore, the information processing device 100 can appropriately generate other information using information generated when an image is generated using a model that generates an image.

[0196] Furthermore, in the information processing device 100 according to the embodiment, the generation unit 132 generates a corresponding image corresponding to the input text, and generates mask information for the target range indicated by the token in the corresponding image based on the attention corresponding to each of the tokens included in the input text used by the diffusion model when generating the corresponding image.

[0197] In this way, the information processing device 100 according to the embodiment generates a corresponding image corresponding to an input text, and generates mask information of a target range indicated by a token in the corresponding image based on attention corresponding to each token included in the input text used by the diffusion model when generating the corresponding image. Therefore, the information processing device 100 can appropriately generate other information using information generated when an image is generated using a model that generates an image.

[0198] In the information processing device 100 according to the embodiment, the acquisition unit 131 acquires designation information that designates one of the tokens included in the input text. The generation unit 132 generates mask information of a target range indicated by the one token in the corresponding image, based on attention corresponding to the one token designated by the designation information among the tokens included in the input text.

[0199] In this way, the information processing device 100 according to the embodiment can generate mask information of a target range indicated by one token in a corresponding image based on attention corresponding to one token specified by the designation information among the tokens included in the input text. Therefore, the information processing device 100 can appropriately generate other information using information at the time of image generation using a model for generating the image.

[0200] In the information processing device 100 according to the embodiment, the acquiring unit 131 acquires designation information that designates multiple tokens among the tokens included in the input text. The generating unit 132 generates mask information of a target range indicated by the multiple tokens in the corresponding image, based on attention corresponding to each of the multiple tokens designated by the designation information among the tokens included in the input text.

[0201] In this way, the information processing device 100 according to the embodiment can generate mask information of the target range indicated by the multiple tokens in the corresponding image based on attention corresponding to each of the multiple tokens specified by the specification information among the tokens included in the input text. Therefore, the information processing device 100 can appropriately generate other information using information at the time of image generation using a model that generates the image.

[0202] Moreover, in the information processing device 100 according to the embodiment, the generation unit 132 generates mask information of a target range corresponding to the logical sum of a plurality of tokens.

[0203] In this way, the information processing device 100 according to the embodiment generates mask information for a target range corresponding to the logical sum of multiple tokens, and can appropriately generate other information using information obtained when generating an image using a model that generates the image.

[0204] Moreover, in the information processing device 100 according to the embodiment, the generation unit 132 generates mask information of a target range corresponding to the logical product of a plurality of tokens.

[0205] In this way, the information processing device 100 according to the embodiment generates mask information for a target range corresponding to the logical product of multiple tokens, and can appropriately generate other information using information obtained when generating an image using a model that generates the image.

[0206] Furthermore, in the information processing apparatus 100 according to the embodiment, the generation unit 132 generates mask information of the target range indicated by the token in the corresponding image based on attention corresponding to each token which is a noun included in the input text.

[0207] In this way, the information processing device 100 according to the embodiment generates mask information for the target range indicated by the token in the corresponding image based on the attention corresponding to each token, which is a noun contained in the input text, and can appropriately generate other information using information obtained when generating an image using a model that generates the image.

[0208] Furthermore, in the information processing apparatus 100 according to the embodiment, the generation unit 132 generates mask information of a target range indicated by a token in a corresponding image, based on attention corresponding to each token that is a verb included in the input text.

[0209] In this way, the information processing device 100 according to the embodiment generates mask information for the target range indicated by the token in the corresponding image based on the attention corresponding to each token, which is a verb contained in the input text, and can appropriately generate other information using information obtained when generating an image using a model that generates the image.

[0210] Furthermore, in the information processing device 100 according to the embodiment, the generation unit 132 generates a corresponding image corresponding to the input text, and generates mask information of the target range indicated by the input text in the corresponding image based on attention at a predetermined stage during the diffusion model generation process.

[0211] In this way, the information processing device 100 according to the embodiment generates a corresponding image corresponding to the input text, and generates mask information of the corresponding image for the target range indicated by the input text based on attention at a predetermined stage during the diffusion model generation process, thereby making it possible to appropriately generate other information using information obtained during image generation using a model that generates an image.

[0212] Moreover, the information processing device 100 according to the embodiment includes a learning unit 133. The learning unit 133 learns a diffusion model, which is a model trained to gradually restore an image indicated by text from a noise image and to restore an image in accordance with attention based on the text.

[0213] In this way, the information processing device 100 according to the embodiment can learn a model that appropriately generates images by training a diffusion model that is trained to restore images according to attention based on text.

[0214] The information processing apparatus 100 according to the embodiment also includes a providing unit 135. The providing unit 135 provides a service using the mask information generated by the generating unit 132.

[0215] In this way, the information processing apparatus 100 according to the embodiment can provide an appropriate service by providing a service using the generated mask information.

[0216] In the information processing device 100 according to the embodiment, the providing unit 135 provides the mask information generated by the generating unit 132 to the user.

[0217] In this way, the information processing apparatus 100 according to the embodiment can provide the user with appropriate services by providing the user with mask information.

[0218] Furthermore, in the information processing device 100 according to the embodiment, the providing unit 135 transmits the mask information generated by the generating unit 132 to the terminal device 10 used by the user.

[0219] In this way, the information processing device 100 according to the embodiment can provide the user with appropriate search results by transmitting mask information to the terminal device 10 used by the user.

[0220] [6. Hardware Configuration] The information processing device 100 and the terminal device 10 according to the above-described embodiments are realized by, for example, a computer 1000 configured as shown in Fig. 11. The following description will be given taking the information processing device 100 as an example. Fig. 11 is a diagram showing an example of a hardware configuration. The computer 1000 is connected to an output device 1010 and an input device 1020, and has a configuration in which a calculation device 1030, a primary storage device 1040, a secondary storage device 1050, an output I / F (Interface) 1060, an input I / F 1070, and a network I / F 1080 are connected via a bus 1090.

[0221] The arithmetic device 1030 operates based on programs stored in the primary storage device 1040 and the secondary storage device 1050, programs read from the input device 1020, and the like, and executes various processes. The arithmetic device 1030 is realized by, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or the like.

[0222] The primary storage device 1040 is a memory device such as RAM (Random Access Memory) that temporarily stores data used by the arithmetic device 1030 for various calculations. The secondary storage device 1050 is a storage device in which data used by the arithmetic device 1030 for various calculations and various databases are registered, and is realized by a ROM (Read Only Memory), an HDD (Hard Disk Drive), an SSD (Solid State Drive), a flash memory, or the like. The secondary storage device 1050 may be an internal storage device or an external storage device. The secondary storage device 1050 may also be a removable storage medium such as a USB memory or an SD (Secure Digital) memory card. The secondary storage device 1050 may also be cloud storage (online storage), a NAS (Network Attached Storage), a file server, or the like.

[0223] The output I / F 1060 is an interface for transmitting information to be output to an output device 1010 that outputs various types of information, such as a display, a projector, a printer, etc., and is realized by a connector conforming to a standard such as USB (Universal Serial Bus), DVI (Digital Visual Interface), or HDMI (High Definition Multimedia Interface), etc. The input I / F 1070 is an interface for receiving information from various input devices 1020, such as a mouse, a keyboard, a keypad, a button, a scanner, etc., and is realized by a USB, etc.

[0224] Furthermore, the output I / F 1060 and the input I / F 1070 may be wirelessly connected to the output device 1010 and the input device 1020, respectively. That is, the output device 1010 and the input device 1020 may be wireless devices.

[0225] The output device 1010 and the input device 1020 may be integrated into one device, such as a touch panel. In this case, the output I / F 1060 and the input I / F 1070 may also be integrated into one device as an input / output I / F.

[0226] The input device 1020 may be a device that reads information from, for example, an optical recording medium such as a CD (Compact Disc), a DVD (Digital Versatile Disc), or a PD (Phase Change Rewritable Disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory.

[0227] The network I / F 1080 receives data from other devices via the network N and sends it to the arithmetic device 1030, and also transmits data generated by the arithmetic device 1030 to other devices via the network N.

[0228] The arithmetic unit 1030 controls the output device 1010 and the input device 1020 via the output I / F 1060 and the input I / F 1070. For example, the arithmetic unit 1030 loads a program from the input device 1020 or the secondary storage device 1050 onto the primary storage device 1040 and executes the loaded program.

[0229] For example, when the computer 1000 functions as the information processing device 100, the arithmetic unit 1030 of the computer 1000 executes a program loaded onto the primary storage device 1040 to realize the functions of the control unit 130. The arithmetic unit 1030 of the computer 1000 may also load a program acquired from another device via the network I / F 1080 onto the primary storage device 1040 and execute the loaded program. The arithmetic unit 1030 of the computer 1000 may also cooperate with the other device via the network I / F 1080 to call and use the functions and data of a program from another program of the other device.

[0230] [7. Other] Furthermore, among the processes described in the above embodiments and modifications, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using known methods. In addition, the information including the processing procedures, specific names, various data, and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified. For example, the various information shown in each drawing is not limited to the information shown in the drawings.

[0231] Furthermore, the components of each device shown in the figure are conceptual functional components and do not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc.

[0232] Furthermore, the above-described embodiments and modifications can be combined as appropriate within the scope of not causing any contradiction in the processing content.

[0233] Furthermore, the above-mentioned "section, module, unit" can be read as "means" or "circuit," etc. For example, an acquisition unit can be read as an acquisition means or an acquisition circuit. [Explanation of symbols]

[0234] 1. Information Processing Systems 100 Information processing device 121 Learning data storage unit 122 Model information storage unit 130 control section 131 Acquisition Department 132 Generation part 133 Learning Department 134 Processing section 135 Provision Department 10 Terminal Equipment 50 Information provision device N Network

Claims

1. an acquisition unit that acquires a diffusion model, which is a model that has been trained to gradually restore an image indicated by text from a noise image, and which has been trained to restore the image in accordance with attention based on the text, and an input text input by a user; a generation unit that generates a corresponding image corresponding to the input text using the diffusion model and the input text acquired by the acquisition unit, and generates mask information of a target range indicated by the input text in the corresponding image based on attention used by the diffusion model when generating the corresponding image; a providing unit that provides the user with information in which the mask information generated by the generating unit, the corresponding image, and a label indicating an object corresponding to the mask information are associated with each other as learning data for learning a model; An information processing device comprising:

2. The generation unit A corresponding image corresponding to the input text is generated, and the mask information of the target range indicated by the token in the corresponding image is generated based on attention corresponding to each token included in the input text used by the diffusion model when generating the corresponding image.

2. The information processing apparatus according to claim 1, wherein:

3. The acquisition unit acquiring designation information that designates one of the tokens included in the input text; The generation unit Generate the mask information of the target range indicated by the one token in the corresponding image based on attention corresponding to the one token specified by the specification information among the tokens included in the input text.

3. The information processing apparatus according to claim 2, wherein:

4. The acquisition unit acquiring designation information that designates a plurality of tokens among the tokens included in the input text; The generation unit generating mask information of the target range indicated by the plurality of tokens in the corresponding image based on attention corresponding to each of the plurality of tokens designated by the designation information among the tokens included in the input text; 4. The information processing apparatus according to claim 3,

5. The generation unit Generate the mask information of the target range corresponding to the logical sum of the plurality of tokens.

5. The information processing apparatus according to claim 4,

6. The generation unit Generate the mask information of the target range corresponding to the logical product of the plurality of tokens.

5. The information processing apparatus according to claim 4,

7. The generation unit generating mask information for the target range indicated by the token in the corresponding image based on attention corresponding to each of the tokens, which are nouns included in the input text; 3. The information processing apparatus according to claim 2, wherein:

8. The generation unit generating mask information of the target range indicated by the token in the corresponding image based on attention corresponding to each of the tokens, which are verbs included in the input text; 3. The information processing apparatus according to claim 2, wherein:

9. The generation unit A corresponding image corresponding to the input text is generated, and the mask information of the target range indicated by the input text is generated from the corresponding image based on attention at a predetermined stage during the generation process of the diffusion model.

2. The information processing apparatus according to claim 1, wherein:

10. a learning unit that learns the diffusion model, which is a model that has been trained to gradually restore the image indicated by the text from the noise image, and which has been trained to restore the image in accordance with attention based on the text; 2. The information processing apparatus according to claim 1, further comprising:

11. The providing unit The mask information generated by the generation unit is transmitted to a terminal device used by the user.

2. The information processing apparatus according to claim 1, wherein:

12. 1. A computer-implemented information processing method, comprising: an acquisition step of acquiring a diffusion model, which is a model trained to gradually restore an image indicated by text from a noise image, and which is trained to restore the image in accordance with attention based on the text, and an input text input by a user; a generation step of generating a corresponding image corresponding to the input text using the diffusion model and the input text acquired in the acquisition step, and generating mask information of a target range indicated by the input text in the corresponding image based on attention used by the diffusion model when generating the corresponding image; a providing step of providing the user with information in which the mask information generated by the generating step, the corresponding image, and a label indicating an object corresponding to the mask information are associated with each other as learning data for learning a model; An information processing method comprising:

13. an acquisition procedure for acquiring a diffusion model, which is a model trained to gradually restore an image indicated by text from a noise image, and which is trained to restore the image in accordance with attention based on the text, and an input text input by a user; a generation step of generating a corresponding image corresponding to the input text using the diffusion model and the input text acquired by the acquisition step, and generating mask information of a target range indicated by the input text in the corresponding image based on attention used by the diffusion model when generating the corresponding image; a providing step of providing the user with information in which the mask information generated by the generating step, the corresponding image, and a label indicating an object corresponding to the mask information are associated with each other as learning data for learning a model; An information processing program characterized by causing a computer to execute the above.

Images