Information processing device, information processing method, and information processing program

Abstract

To appropriately generate other information using information when an image is generated using an image generation model.SOLUTION: An information processing device comprises an acquisition unit and a generation unit. The acquisition unit acquires a target image, which is an image meeting a prescribed condition, a corresponding text, which is a text associated with the target image, and a diffusion model, which is a model learned so as to restore the target image step by step from a noise image and the corresponding text and also learned so as to restore the target image according to an attention based on the corresponding text. The generation unit generates mask information for masking a target range indicated by the corresponding text in the target image on the basis of the attention used when the diffusion model generated the target image using the diffusion model and corresponding text acquired by the acquisition unit.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing method, and an information processing program.

Background Art

[0002] Conventionally, techniques for generating various types of information related to images have been provided. For example, there is a technique for generating information such as mask information (mask image) that identifies the background in an image and masks that location.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Non-Patent Documents

[0005] However, the conventional technologies described above have room for improvement. For example, the conventional technologies require a dedicated model to identify the background and generate mask information, and without such a dedicated model, it is difficult to generate the desired information. Therefore, it is desirable to appropriately generate other information, such as mask information, using the information processed by the image generation model.

[0006] This application has been made in view of the above, and aims to provide an information processing device, an information processing method, and an information processing program that appropriately generate other information using information generated during image generation using an image generation model. [Means for solving the problem]

[0007] The information processing device according to the present invention is characterized by comprising: an acquisition unit that acquires a target image which is an image satisfying predetermined conditions, a corresponding text which is text associated with the target image, and a diffusion model which is a model that has been trained to gradually reconstruct the target image from a noise image and the corresponding text, and which is a model that reconstructs the target image in accordance with attention based on the corresponding text; and a generation unit that uses the diffusion model and the corresponding text acquired by the acquisition unit to generate mask information that masks the target range indicated by the corresponding text in the target image, based on the attention used by the diffusion model when generating the target image. [Effects of the Invention]

[0008] According to one embodiment, the effect is achieved that other information can be appropriately generated using the information generated during image generation using an image generation model. [Brief explanation of the drawing]

[0009] [Figure 1] Figure 1 shows an example of information processing according to the embodiment. [Figure 2] Figure 2 shows another example of information processing according to the embodiment. [Figure 3] Figure 3 shows an example of the configuration of an information processing system according to the embodiment. [Figure 4] Figure 4 shows an example of the configuration of an information processing device according to the present invention. [Figure 5] Figure 5 shows an example of a model information storage unit according to the embodiment. [Figure 6] Figure 6 is a flowchart showing an example of information processing according to the embodiment. [Figure 7] Figure 7 is a flowchart showing an example of information processing according to the embodiment. [Figure 8] Figure 8 is a flowchart showing an example of information processing according to the embodiment. [Figure 9] Figure 9 shows an example of the model configuration. [Figure 10] FIG. 10 is a diagram showing an example of the configuration of the main part of the model. [Figure 11] FIG. 11 is a diagram showing an example of the hardware configuration.

Embodiment for Implementing the Invention

[0010] Hereinafter, embodiments for implementing the information processing apparatus, information processing method, and information processing program according to the present application (hereinafter referred to as "embodiments") will be described in detail with reference to the drawings. Note that the information processing apparatus, information processing method, and information processing program according to the present application are not limited by this embodiment. Also, in each of the following embodiments, the same parts are denoted by the same reference numerals, and duplicate descriptions are omitted.

[0011] (Embodiment) 〔1. Information Processing〕 Hereinafter, an example of the information processing according to the embodiment will be described using FIG. 1. FIG. 1 is a diagram showing an example of the information processing according to the embodiment. Specifically, FIG. 1 shows an example of information processing including the learning process of the model according to the embodiment and the image generation process using the model. In FIG. !, an example will be described in which an information processing apparatus 100 (see FIG. 4) learns (generates) a diffusion model M1 (hereinafter also referred to as "model M1"), which is a diffusion model for generating an image from a noise image, and generates an image using the generated model M1. Also, the generation of other information such as mask information using the information during the image generation process using model M1 will be described after explaining the outline of the model in FIG. 1.

[0012] 〔1-1. Example of Learning Process〕 First, an example of the learning process of model M1 will be described using FIG. 1. First, the data (learning data) used for the learning process of model M1 will be described.

[0013] In Figure 1, the information processing device 100 acquires data (training data) to be used in the training process of model M1. For example, the information processing device 100 acquires the training data for model M1 from the training data storage unit 121 (see Figure 4). If no training data is registered in the training data storage unit 121, the information processing device 100 may request training data from an external device such as the information providing device 50 (see Figure 3), and register the data received from the external device as training data (also called "training data") in the training data storage unit 121.

[0014] Figure 1 shows an example of data used for training model M1, namely image group IG1, which includes multiple step images in which noise is progressively added to a predetermined image. The information processing device 100 trains model M1 using training data including image group IG1 shown in Figure 1.

[0015] Image group IG1 includes image P10, which is an image (training image) in which a predetermined object is captured. Image P10 is an image in which a sofa and a person (woman) are captured as the predetermined object. Image P10 is at noise level #0, meaning it is an image without noise.

[0016] Furthermore, the image group IG1 includes image P11, which is image P10 with noise added to it. Image P11 is image P10 with one level of noise added to it. Image P11 is an image with noise added at level #1, i.e., one level of noise added to it. For example, image P11 is generated by a noise addition process (process SN1 in Figure 1) that adds noise to image P10.

[0017] Furthermore, the image group IG1 includes image P12, which is image P11 with noise added to it. Image P12 is image P11 with one additional noise added to it. Image P12 is an image with noise added in stage #2, i.e., two stages of noise added. For example, image P12 is generated by a noise addition process (process SN2 in Figure 1) that adds noise to image P11.

[0018] Furthermore, the image group IG1 includes image P13, which is image P12 with noise added to it. For example, the noise addition process (process SN3 in Figure 1) that adds noise to image P12 generates image #3 (image P13), which has noise added in three stages. Note that the intermediate processes are omitted because the noise is repeated.

[0019] Furthermore, the image group IG1 includes image P1K-2, which is an image to which noise has been added to image P1K-3, which has K-3 levels of noise added. Here, K is any number (value), such as 30 or 1000. For example, a noise addition process (process SNK-2 in Figure 1) that adds noise to image P1K-3 generates an image with K-2 levels of noise added, which is image #K-2 (image P1K-2).

[0020] Furthermore, the image group IG1 includes image P1K-1, which is image P1K-2 with noise added to it. Image P1K-1 is image P1K-2 with one additional level of noise added to it. Image P1K-1 has a noise addition level of #K-1, meaning it has K-1 levels of noise added. For example, image P1K-1 is generated by a noise addition process (process SNK-1 in Figure 1) that adds noise to image P1K-2.

[0021] Furthermore, the image group IG1 includes the noisy image P1K. For example, image P1K is an image in which one more level of noise has been added to image P1K-1. Image P1K is an image in which the level of noise addition is level #K, that is, an image in which K levels of noise have been added. For example, image P1K is generated by a noise addition process (process SNK in Figure 1) that adds noise to image P1K-1. Processes SN1 to SNK in Figure 1 are processes that correspond to the diffusion process (forward process) in the diffusion model.

[0022] For example, each image in the image group IG1 may be generated by a process (noise addition process) that adds Gaussian noise to the data, gradually transitioning from real data to complete Gaussian noise. For example, if K is 1000, each image in the image group IG1 is generated by mixing the data and noise while changing the ratio so that complete noise is achieved after 1000 noise addition processes. Note that the case where K is 1000, i.e., the number of noise addition processes is 1000, is just one example, and the number of noise addition processes may be less than 1000, such as 32 or 1, or more than 1000. In addition, each image in the image group IG1 may be generated by the information processing device 100, or by an external device such as the information providing device 50.

[0023] As shown above, the image group IG1 in Figure 1 includes multiple images such as images P10, P11, P12, ... P1K-2, P1K-1, P1K, etc. Below, we will explain, as an example, the case in which the information processing device 100 learns model M1 using the above-described image group IG1. Note that the above is merely an example, and the images included in the image group IG1 may be any images. For example, the images included in the image group IG1 are not limited to real images such as captured images (photographs) of subjects (objects) actually captured by imaging devices such as cameras, but may also be images generated by information processing using an image generation model, etc. Also, for example, image P1K may be a noise image such as so-called white noise.

[0024] From here, we will explain the details of the learning process example. In Figure 1, the information processing device 100 performs a learning process to learn model M1 using the image group IG1, etc. (step S1). The information processing device 100 takes an image containing noise (noise image) as input and learns model M1 to output (predict) an image obtained by removing one level of noise from the noise image, which is the previous step. For example, the information processing device 100 learns model M1 which takes a noise image, text (character information) indicating the content of the image to be generated (e.g., the object or phenomenon to be included in the image), and information indicating which level of noise the noise image is at (level information) as input. The object or phenomenon may be collectively referred to as "object". Note that if the level information is used as information managed within model M1 (internal information), it does not need to be included in the input of model M1.

[0025] The text input to Model M1 can be in any form as long as it describes the content of the image to be generated. The text may be a string of words that describes an object. In this case, the text may be a word that describes a single object, such as a bear or a sofa. Alternatively, the text may be a string of words that describe each of multiple objects. In this case, the text may be a string of words that describe each of multiple objects, such as a bear, a bird, a sofa and a woman, as well as particles such as coordinating particles. Furthermore, the text may be a string of words that has meaning as a whole (a sentence). In this case, the text may be a sentence that describes the content that you want to express in the image, such as a bear watching a flying bird or a woman in a dress sitting on a sofa. Note that the above examples of input to Model M1 are just a few examples, and Model M1 may accept any information as input.

[0026] In Figure 1, the information processing device 100 trains model M1 to generate an image in which one level of noise has been removed from a noisy image of one level. In Figure 1, the information processing device 100 trains model M1 using text describing the content of image P10 (also called "training text") and level information (values ​​indicating the level to be processed).

[0027] For example, the information processing device 100 takes image P1K, training text (e.g., text TX1), and stage information as input and performs a training process (process SR1 in Figure 1) to train model M1 to output image P1K-1. For example, text TX1 is the sentence, "A woman wearing a dress is sitting on a sofa." The stage information may be a value corresponding to time t in Figure 9. In this case, the stage information may be information such as the denoising stage "0" or the noise stage "K". For example, the stage information may be the number of iterations, for example, information indicating which it is the iteration (0, 1, etc.). In this way, the information processing device 100 trains model M1 to generate image P1K-1, which is a noise image at stage #K-1 of the previous process, from image P1K, which is a noise image at stage #K.

[0028] Furthermore, the information processing device 100 takes image P1K-1, a noise image at stage #K-1, training text, and stage information as inputs and performs a learning process (process SR2 in Figure 1) to train model M1 to output image P1K-2, a noise image at stage #K-2. Similarly, the information processing device 100 takes image P1K-2, a noise image at stage #K-2, training text, and stage information as inputs and performs a learning process (process SR3 in Figure 1) to train model M1 to output image P1K-3, a noise image at stage #K-3. Note that the processing between process SR3 and process SRK-2 in Figure 1 is the same as the process described above, taking a noise image as input and training the model to output the image one stage before the noise was added to that noise image, so a detailed explanation is omitted.

[0029] The information processing device 100 takes image P13, a noise image from stage #3, training text, and stage information as input and performs a training process (process SRK-2 in Figure 1) to train model M1 to output image P12, a noise image from stage #2. The information processing device 100 takes image P12, a noise image from stage #2, training text, and stage information as input and performs a training process (process SRK-1 in Figure 1) to train model M1 to output image P11, a noise image from stage #1.

[0030] The information processing device 100 then takes image P11, which is a noised image of stage #1, training text, and stage information as input and performs a learning process (process SRK in Figure 1) to train model M1 to output image P10, which is stage #0, i.e., an image without noise. Processes SR1 to SRK in Figure 1 correspond to the generative process (reverse process) in the diffusion model. In Figure 1, the information processing device 100 trains model M1 using a group of images IG1 including images P10, P11, P12, ... P1K-2, P1K-1, P1K, etc.

[0031] In this way, the information processing device 100 learns model M1, which is a model that generates data by taking a noisy image as input and gradually removing the noise. In Figure 1, the information processing device 100 learns model M1, which generates data (also called a "generated image") that reflects the content indicated by the text, by taking a noisy image, text, and step information as input and gradually removing the noise from the noisy image.

[0032] In Figure 1, the information processing device 100 learns a model M1 that generates a generated image showing the content of the text, for example, a situation (state) of a woman wearing a dress sitting on a sofa, by gradually removing noise from the noisy image. In this way, the information processing device 100 learns a model M1 that transforms the noisy image to resemble the training image based on the text.

[0033] Note that the learning process using the image group IG1 shown in Figure 1 is only a part of the learning of model M1, and the information processing device 100 learns model M1 using training data that includes various image groups in addition to the image group IG1. The process shown in Figure 1 is merely an example, and the information processing device 100 may learn model M1 based on learning methods related to diffusion models disclosed in Non-Patent Documents 2, 3, etc.

[0034] In this way, the information processing device 100 learns a diffusion model that transforms a noisy image to approximate a training image based on a training image containing a predetermined object and text indicating the predetermined object. As a result, the information processing device 100 learns to stepwise reconstruct the image indicated by the text from the noisy image and learns Model M1, which is a diffusion model that has been trained to reconstruct the image in accordance with attention based on the text.

[0035] In the example described above, the information processing device 100 inputs the image along with text describing the content of the image to the diffusion model and learns the diffusion model. For example, if text describing the content of the image is also collected when the image is collected, the text is also known, so the information processing device 100 uses that data as training data to learn a model such as model M1 using self-supervised learning. Note that if another device (learning device) learns the diffusion model or other models, the information processing device 100 does not need to learn the diffusion model. If the information processing device 100 does not learn the diffusion model, the information processing device 100 may acquire model M1, which is a diffusion model, by receiving it from the learning device.

[0036] [1-2. Example of generation process] Next, we will explain an example of image generation processing using Model M1. In Figure 1, for illustrative purposes, we will explain the case where the text used to generate the generated image (also called "corresponding text") is text TX1, but the corresponding text is not limited to text TX1; it can be any image. For example, the information processing device 100 receives a specification from the user that text TX1 should be used to generate the generated image. Thus, in the following explanation, we will explain the case where text TX1 is text entered by the user (also called "input text") as an example.

[0037] The information processing device 100 takes a noise image, text, and step information as input and performs a generation process to generate a generated image (step S11). For example, the noise image NZ1 may be the same noise image as image P1K, or it may be a different noise image. In Figure 1, the information processing device 100 takes the noise image NZ1, text TX1, and step information as input information IN1 to the first model M1 and generates image P20 by repeatedly generating images using model M1.

[0038] For example, the information processing device 100 generates image P20 by repeating the input to model M1 K times. For example, if K is 1000, the information processing device 100 takes the noise image NZ1, text TX1, and the image output by model M1 (which has step information as input) as input and repeats the input to model M1 999 times.

[0039] For example, in the first iteration, the information processing device 100 inputs the noise image NZ1, text TX1, and step information indicating that it is the first iteration to model M1, and outputs image #1 from model M1. Then, in the second iteration, the information processing device 100 inputs image #1, text TX1, and step information indicating that it is the second iteration to model M1, and outputs image #2 from model M1. In the third iteration, the information processing device 100 inputs image #2, text TX1, and step information indicating that it is the third iteration to model M1, and outputs image #3 from model M1.

[0040] Similarly, in the 4th to K-1th iterations, the information processing device 100 inputs images #3 to #K-2, text TX1, and corresponding step information to model M1, and outputs images #4 to #K-1 from model M1. Then, in the Kth iteration, the information processing device 100 inputs image #K-1, text TX1, and step information indicating that it is the Kth iteration to model M1, and outputs image P20 from model M1.

[0041] As a result, the information processing device 100 generates image P20, which represents the content of text TX1, by gradually removing noise from the noisy image. In Figure 1, the information processing device 100 generates image P20, which is a generated image representing the situation described in text TX1, where a woman wearing a dress is sitting on a sofa, by gradually removing noise from the noisy image using model M1. In this way, the information processing device 100 generates image P20, which represents the content of text TX1, in the same way as image P10, by gradually removing noise from the noisy image based on text TX1.

[0042] [1-3. Examples of Information Generation] The above-described process makes it possible to generate images using a diffusion model that generates images showing the content of text from noise images. The information generated during the diffusion model processing is useful for purposes other than generating images, and can be used to generate other types of information. Therefore, the information processing device 100 uses the information generated during the diffusion model processing described above to generate other information such as mask information. First, before explaining examples of generating other types of information, we will explain an example of the configuration of the diffusion model M1 using Figure 9. Figure 9 is a diagram showing an example of the model configuration.

[0043] The example structure of the model shown in Figure 9 illustrates one example of the structure of Model M1. In this case, Model M1 has a structure based on U-net. However, the U-net-based structure is merely one example, and Model M1 is not limited to a U-net-based structure; any structure can be adopted as long as the desired processing is possible. For example, Model M1 may have a structure based on a method other than U-net, such as a Transformer. Since the U-net-based structure is known, a detailed explanation will be omitted, but a brief overview will be given below.

[0044] In Figure 9, Model M1, which has a U-net-based structure, has a structure that includes multiple ResBlocks, which are first-type blocks, and AttnBlocks, which are second-type blocks. For example, a ResBlock, which is a first-type block, is a residual block that has a combination of a convolutional layer and a SkipConnection. An AttnBlock, which is a second-type block, is a block that has an attention function, and its structure will be described later.

[0045] In the model shown in Figure 9, the left side corresponds to the input side, and the right side corresponds to the output side. The straight lines and arrows connecting each part indicate the concatenation relationship between the parts. For example, model M1 applies a first type block (ResBlock) and a second type block (AttnBlock) to the input. In the first half (the left side of Figure 9 (input side)), it reduces the resolution of the input and increases the number of channels, and in the second half (the right side of Figure 9 (output side)), it reduces the number of channels and increases the resolution.

[0046] The model structure shown in Figure 9 has a structure in which pairs of Type 1 blocks and Type 2 blocks are connected in multiple layers. For example, in Figure 9, "Z t The input section corresponding to "t" corresponds to the section where an image (noise image) corresponding to time t is input. For example, the input section corresponding to "t" in Figure 9 corresponds to the section where information indicating the stage of processing is input (stage information input section). The stage information input section is connected to each of the first type block and the second type block, and provides stage information to each block.

[0047] For example, "TextEncoder" in Figure 9 corresponds to the encoding unit, which performs the process of encoding (encoding) the input text. The encoding unit is linked to each of the second type blocks and provides text information (encoding information) to each of the second type blocks. Thus, in the model shown in Figure 9, the text information encoded by the encoding unit indicated by "TextEncoder" is input to each of the second type blocks.

[0048] The AttnBlock, a second type of block with attention functionality, has the structure shown in Figure 10. Figure 10 is a diagram showing an example of the configuration of the main parts of the model. Specifically, Figure 10 is a diagram showing an example of the configuration of a second type of block.

[0049] In the second type of block shown in Figure 10, the left side corresponds to the input side and the right side corresponds to the output side. As shown in Figure 10, in the second type of block (AttnBlock), information from the encoding section is used in the cross-attention section, which has the function of cross-attention and is labeled "Cross-Attn".

[0050] The second type of block shown in Figure 10 includes, in addition to the cross-attention section, a section with a self-attention function labeled "Self-Attn" and a section with a dropout function labeled "Dropout". The configuration within the second type of block is not limited to that shown in Figure 10; any configuration with a cross-attention section that uses information from text can be adopted, and a detailed explanation is omitted.

[0051] For example, the cross-attention unit calculates the cross-attention between text information and image information. The cross-attention unit obtains the cross-attention result according to the content of the text. In this way, the cross-attention unit generates information (also called "attention information") that indicates the result of the cross-attention between text information and image information.

[0052] Here, using Figure 2, we show an example of the attention information generated at each stage. Figure 2 is a diagram showing another example of the information processing according to the embodiment. In Figure 2, for explanatory purposes, we show as an example the case in which image P20 is generated from image P2K (corresponding to the noise image NZ1 in Figure 1), which is the noise image in Figure 1.

[0053] Text TX1 is divided into token TK1, which is labeled "Token #1" in Figure 2, and token TX2, which is labeled "Token #2" in Figure 2. Note that when explaining tokens TK1 and TK2 without distinguishing between them, they will be referred to as "Token TK".

[0054] In Figure 2, token TK1 is "sofa" and token TX2 is "woman wearing a dress". For example, tokens TK1 and TX2 correspond to each of the multiple elements that make up text TX1. For example, tokens TK1 and TX2 are strings of characters contained in text and are elements that indicate the content (meaning) of a certain object or phenomenon. For example, a token is the unit in which attention information is generated in cross-attention.

[0055] In Figure 2, Model M1 generates attention information corresponding to each token at each stage of processing. For example, in the process of generating image P2K-1 (image #1), Model M1 generates attention information for the first processing step in response to inputs of image P2K, text TX1, and stage information indicating that it is the first processing step.

[0056] In Figure 2, Model M1 generates attention information A1K for token TK1 (token #1) as attention information in the first processing. Attention information A1K indicates the region in the image (image P2K) that corresponds to token TK1.

[0057] Each piece of attention information, such as attention information A1K shown in Figure 2, is, for example, an attention map that indicates the degree of association between each region (pixel, etc.) of an image and the object indicated by the token. In the example in Figure 2, the darker the hatching in the attention information, the higher the association between the region (pixel, etc.) and the object indicated by the token in the corresponding image. For example, the darker the hatching in the attention information, the more likely it is that the object indicated by the token is contained in the corresponding region. Conversely, the lighter the hatching in the attention information, the lower the association between the region (pixel, etc.) and the object indicated by the token in the corresponding image. For example, the lighter the hatching, the more likely it is that the object indicated by the token is not contained in the corresponding region. For example, the darker the hatching in the attention information, the larger the value at that location, and the lighter the hatching, the smaller the value at that location.

[0058] Furthermore, in Figure 2, Model M1 generates attention information A2K for token TK2 (token #2) as attention information in the first processing. Attention information A2K is information indicating the region in the image (e.g., image P2K) that corresponds to token TK2.

[0059] Furthermore, in the process of generating image P2K-2 (image #2) in response to inputs of image P2K-1 (image #1), text TX1, and stage information indicating that it is the second processing step, model M1 generates attention information for the second processing step.

[0060] In Figure 2, Model M1 generates attention information A1K-1 for token TK1 as attention information in the second processing. Attention information A1K-1 is information indicating the region in the image (e.g., image P2K-1) that corresponds to token TK1.

[0061] Furthermore, in Figure 2, Model M1 generates attention information A2K-1 for token TK2 as attention information in the second processing. Attention information A2K-1 is information indicating the region in the image (e.g., image P2K-1) that corresponds to token TK2.

[0062] Furthermore, in the 3rd to K-3rd iteration, Model M1 similarly generates attention information for each of the 3rd to K-3rd iterations in the process of generating each of the images #3 to #K-3, in response to the input of images #2 to #K-4, text TX1, and step information corresponding to the processing.

[0063] Furthermore, in the process of generating image P22 (image #K-2), model M1 generates attention information for the K-2nd processing step in response to inputs of image P23 (image #K-3), text TX1, and step information indicating that it is the K-2nd processing step.

[0064] In Figure 2, Model M1 generates attention information A13 for token TK1 as attention information during the (K-2)th processing. Attention information A13 indicates the region in the image (e.g., image P23) corresponding to token TK1. In Figure 2, attention information A13 is assumed to be the attention information that most appropriately indicates the region in the image where the object indicated by token TK1 is located.

[0065] Furthermore, in Figure 2, Model M1 generates attention information A23 for token TK2 as attention information during the (K-2)th processing. Attention information A23 indicates the region in the image (e.g., image P23) that corresponds to token TK2. In Figure 2, attention information A23 is assumed to be the attention information that most appropriately indicates the region in the image where the object indicated by token TK2 is located.

[0066] Furthermore, in the process of generating image P21 (image #K-1), model M1 generates attention information for the K-1th processing step in response to inputs of image P22 (image #K-2), text TX1, and step information indicating that it is the K-1th processing step.

[0067] In Figure 2, Model M1 generates attention information A12 for token TK1 as attention information during the (K-1)th processing. Attention information A12 indicates the region in the image (e.g., image P22) that corresponds to token TK1.

[0068] Furthermore, in Figure 2, Model M1 generates attention information A22 for token TK2 as attention information during the K-1th processing. Attention information A22 is information indicating the region in the image (e.g., image P22) that corresponds to token TK2.

[0069] Furthermore, in the process of generating image P20 (image #K) in response to inputs of image P21 (image #K-1), text TX1, and step information indicating that it is the Kth processing step, model M1 generates attention information for the Kth processing step.

[0070] In Figure 2, Model M1 generates attention information A11 for token TK1 as attention information during the Kth processing step. Attention information A11 is information indicating the region in the image (e.g., image P21) that corresponds to token TK1.

[0071] Furthermore, in Figure 2, Model M1 generates attention information A21 for token TK2 as attention information during the Kth processing step. Attention information A21 is information indicating the region in the image (e.g., image P21) that corresponds to token TK2.

[0072] Thus, in Model M1, attention information is generated during the process of generating the generated image, showing the result of cross-attention between the text information and the image information. The information processing device 100 then uses the attention information generated during the process of generating the generated image to generate the following information.

[0073] In the following, when describing attention information A1K, A1K-1, ... A12, A11, etc., corresponding to token TK1 (token #1) without distinction, it will be referred to as attention information A1. Similarly, when describing attention information A2K, A2K-1, ... A22, A21, etc., corresponding to token TK2 (token #2) without distinction, it will be referred to as attention information A2. Furthermore, when describing all of attention information A1 and attention information A2, i.e., attention information A1K, A2K, A1K-1, A2K-1, ... A12, A22, A11, A21, etc., in Figure 2 without distinction, it will be referred to as attention information A.

[0074] [1-3-1. Example of generating mask information] First, the following describes a first generation example in which the information processing device 100 generates mask information using attention information A, with reference to Figure 2. Note that explanations of points similar to those described above will be omitted as appropriate.

[0075] In Figure 2, the information processing device 100 uses model M1 and text TX1 to generate image P20, which is a corresponding image to text TX1, and generates mask information for the target range indicated by text TX1 within image P20 based on attention information A used by model M1 when generating image P20. The information processing device 100 generates mask information for the target range indicated by text TX1 within image P20 based on attention information A at a predetermined stage during the generation process of image P20, which is an image generated by model M1.

[0076] For example, the information processing device 100 generates mask information for the target range indicated by token TK in image P20, based on attention information A corresponding to each token contained in text TX1 used by model M1 when generating image P20. The information processing device 100 obtains specification information from the user specifying one of the tokens TX1 and TX2 contained in text TX1, and generates mask information for the target range indicated by that token in image P20, based on attention information A corresponding to the token indicated by the specification information.

[0077] The following example describes the case where both tokens TX1 and TX2 contained in text TX1 are specified by the user. The information processing device 100 obtains specification information from the user specifying each of the multiple tokens TX1 and TX2 contained in text TX1, and generates mask information for the target range indicated by each of the multiple tokens TX1 and TX2 in image P20 based on attention information A1 and A2 corresponding to each of the multiple tokens TX1 and TX2 indicated by the specification information.

[0078] For example, the information processing device 100 generates mask information for the target range indicated by token TK1 in image P20 based on attention information A1 corresponding to token TK1 contained in text TX1 used by model M1 when generating image P20. For example, the information processing device 100 generates mask information for the target range indicated by token TK1 in image P20 based on attention information A1 at a stage determined based on information stored in the storage unit 120. For example, the information processing device 100 generates mask information for the target range indicated by token TK1 in image P20 based on attention information A1 at a pre-set stage (specified stage).

[0079] Furthermore, the attention information used when generating mask information is not limited to one piece of attention information; multiple pieces of attention information may be used. In this case, the information processing device 100 may use attention information (synthesized attention information) calculated (generated) using an arbitrary calculation method such as a weighted average, targeting multiple pieces of attention information corresponding to each of the multiple processing counts (timings). Below, we will show the case in which one piece of attention information is selected and used that most appropriately indicates the region where the target indicated by each token TK is located.

[0080] In Figure 2, since attention information A13 in the K-2 processing is the attention information that most appropriately indicates the region in the image where the object indicated by token TK1 is located, the information processing device 100 generates mask information for the target range indicated by token TK1 in image P20 based on attention information A13. For example, the information processing device 100 generates mask information that distinguishes between the range in image P20 that includes the object "sofa" indicated by token TK1 (target range) and the other range (also called the "non-target range") based on attention information A13.

[0081] For example, based on attention information A13, the information processing device 100 generates mask information that masks the area of ​​image P20 that is outside the target range of token TK1. For example, based on attention information A13, the information processing device 100 generates a segmentation mask as mask information that shows the segmentation of the object "sofa" indicated by token TK1 in image P20. For example, the information processing device 100 generates mask information that shows the range of the attention information A13 in which the value satisfies a predetermined condition as the segmentation of the object "sofa" indicated by token TK1. For example, the information processing device 100 generates mask information that shows the range (pixels, etc.) in which the value is greater than or equal to a predetermined value as the segmentation of the object "sofa" indicated by token TK1. In Figure 2, the information processing device 100 generates mask information that shows the hatched range of the attention information A13 as the segmentation of the object "sofa" indicated by token TK1. For example, the information processing device 100 generates mask information that masks the range of attention information A13 where the value is less than a predetermined value as an excluded range.

[0082] For example, the information processing device 100 generates mask information for the target range indicated by token TK2 in image P20, based on attention information A2 corresponding to token TK2 contained in text TX1 used by model M1 when generating image P20. For example, the information processing device 100 generates mask information for the target range indicated by token TK2 in image P20, based on attention information A2 at a pre-set stage (specified stage).

[0083] In Figure 2, since attention information A23 in the K-2 processing step is the attention information that most appropriately indicates the region in the image where the object indicated by token TK2 is located, the information processing device 100 generates mask information for the target range indicated by token TK2 in image P20 based on attention information A23. For example, based on attention information A23, the information processing device 100 generates mask information that distinguishes between the range in image P20 that includes the object "woman wearing a dress" indicated by token TK2 (target range) and the range that is not targeted.

[0084] For example, based on attention information A23, the information processing device 100 generates mask information that masks the area of ​​image P20 that is outside the target range of token TK2. For example, based on attention information A23, the information processing device 100 generates a segmentation mask as mask information that shows the segmentation of image P20 related to the target "woman wearing a dress" indicated by token TK2. For example, the information processing device 100 generates mask information that shows the range of attention information A23 where the value satisfies a predetermined condition as the segmentation of the target "woman wearing a dress" indicated by token TK2. In Figure 2, the information processing device 100 generates mask information that shows the hatched range of attention information A23 as the segmentation of the target "woman wearing a dress" indicated by token TK2. For example, the information processing device 100 generates mask information that masks the range of attention information A23 where the value is less than a predetermined value as the area outside the target range.

[0085] As described above, the information processing device 100 can appropriately generate other information using the information generated during image generation using an image generation model. For example, the information processing device 100 uses text, a target image to be generated (also called the "target image"), and images with noise added to the target image in stages to learn a diffusion model (model M1, etc.) in each iteration to generate an image that is one step closer to the target image based on the input noise image and text. In this way, the information processing device 100 can learn a diffusion model (model M1, etc.) to generate an image that the text indicates in stages when an image and text are input.

[0086] For example, the information processing device 100 receives text and token specifications from the user. The information processing device 100 generates images corresponding to the text using a diffusion model (model M1, etc.). The information processing device 100 generates mask information from attention information corresponding to each token generated during the image generation process using the diffusion model (model M1, etc.). The information processing device 100 provides various services using the mask information and the generated images.

[0087] The information processing device 100 performs various operations as described above. For example, the information processing device 100 generates mask information that extracts the image portion corresponding to the text. For example, if the token is a noun, the information processing device 100 generates mask information that identifies the range (target range) that includes the object corresponding to that noun.

[0088] For example, if the token is a verb, the information processing device 100 generates mask information that identifies the range (target range) that includes the event corresponding to that verb. This allows the information processing device 100 to visualize the event indicated by a verb such as "see" and generate mask information that identifies its range (target range). For example, the information processing device 100 can dynamically communicate this range to visually impaired individuals using embossed information generated based on the mask information.

[0089] For example, when text specifying a state (behavioral state) such as an action or pose (posture, gesture, etc.) is used, the information processing device 100 generates mask information that identifies the range (target range) that includes events corresponding to the behavioral state indicated by that text. In this way, the information processing device 100 generates mask information that identifies the range (target range) that includes events corresponding to the "action XX" indicated by text such as "a person performing action XX".

[0090] Furthermore, the information processing device 100 can generate mask information that combines multiple tokens. For example, if the tokens "bear" and "bird" are connected by a parallel particle in a logical OR (logical OR), such as "bear and bird," the information processing device 100 generates mask information (logical OR mask information) that identifies both the first target range containing the object corresponding to the token "bear" and the second target range containing the object corresponding to the token "bird." In this case, the information processing device 100 generates mask information that distinguishes the first target range and the second target range from other ranges. The information processing device 100 may also generate mask information that distinguishes both the first target range and the second target range by adding information that distinguishes each of the first and second target ranges.

[0091] Furthermore, if the tokens "flying" and "car" are in a logical AND relationship, such as "a car flying in the sky," the information processing device 100 generates mask information (logical AND mask information) that identifies the range of objects that include the object corresponding to the combined token "a car flying in the sky." Note that the above is merely an example, and the information processing device 100 may generate desired information through any processing. In this way, the information processing device 100 can appropriately generate other information using the information obtained during image generation using an image generation model.

[0092] The information processing device 100 may generate mask information using arbitrary information. For example, the information processing device 100 may obtain cross-attention results (attention information) of multiple different resolutions based on the position of the second type of block using the cross-attention unit shown in Figure 10. In this case, the information processing device 100 may select attention information according to the resolution and generate mask information from the selected attention information.

[0093] For example, the information processing device 100 may generate mask information for the target area indicated by token TK1 in image P20 based on attention information A1 corresponding to the resolution determined based on information stored in the storage unit 120. For example, the information processing device 100 may generate mask information for the target area indicated by token TK1 in image P20 based on attention information A1 that is closest to a preset resolution (specified resolution). In addition, the information processing device 100 may also generate mask information for the target area indicated by token TK2 in image P20 based on attention information A2 corresponding to the resolution, similar to the case of token TK1.

[0094] Furthermore, the information processing device 100 provides services using the generated mask information. For example, the information processing device 100 transmits the generated mask information to the terminal device 10 used by the user. For example, the information processing device 100 transmits to the terminal device 10 used by the user information that associates the mask information generated using the information generated during the image generation process in Model M1 with the generated image generated in Model M1.

[0095] For example, the information processing device 100 transmits to the user's terminal device 10 information that associates the generated image produced by model M1, the mask information generated using the information from the image generation process, and a label indicating the target corresponding to the mask information. For example, the information processing device 100 transmits to the user's terminal device 10 information that associates the generated image produced by model M1, the mask information, and a label indicating the target corresponding to the mask information, as training data.

[0096] Through the processing described above, the information processing device 100 can appropriately generate arbitrary information, such as training data. For example, the information processing device 100 can automatically generate a segmentation mask corresponding to any word in the input text. For example, one application is to improve accuracy in an image classification task by using the generated image. Therefore, by using the information generated by the information processing device 100 as training data, images with masks can be generated, making it applicable to classification tasks, detection tasks, segmentation tasks, etc.

[0097] Furthermore, the information processing system 1 (see Figure 3) may provide a UI (User Interface) that searches for an object corresponding to the part of an image specified (touched) by the user. In this case, the information processing system 1 performs a process to search for an object in the part of the image specified by the user, thereby providing information indicating the object contained in the touched part of the pixel, for example, for young children. Also, as mentioned above, the information processing system 1 may provide a service that assists in the creation of embossed books to help visually impaired people understand the shapes of animals, etc. Furthermore, in the case of a book in which the embossing generated by the information processing system 1 can be manipulated in real time, the system can provide information (words, etc.) indicating the object corresponding to the part of the image specified by the user. In addition, the information processing system 1 can identify a region in an image containing many objects by specifying a word. For example, the information processing system 1 can identify a region in an image that contains a person performing a specific action.

[0098] The process described above is merely one example, and the information processing device 100 may generate other information using various types of information. For example, the images used for training model M1 may be images that satisfy predetermined conditions. A second generation example in which the information processing device 100 generates information based on images that satisfy predetermined conditions will be described. Note that explanations of points similar to those described above will be omitted as appropriate.

[0099] For example, model M1 may be a diffusion model that has been trained to gradually reconstruct the target image in response to attention based on the corresponding text, using a target image which is an image that satisfies predetermined conditions, text associated with the target image (corresponding text), and a noise image. In this case, the information processing device 100 may train model M1 to gradually reconstruct the target image in response to attention based on the corresponding text, using a target image which is an image that satisfies predetermined conditions, text associated with the target image, and a noise image.

[0100] For example, an image that satisfies the specified conditions may be a real image. For example, an image that satisfies the specified conditions may be an image (real image) captured by an imaging device such as a camera of a specified object. For example, model M1 may be a diffusion model that has been trained to reconstruct a real image step by step in accordance with attention based on the corresponding text, using a real image, corresponding text that describes the content of the real image, and a noise image.

[0101] In this case, the information processing device 100 may learn model M1 to reconstruct the real image step by step according to attention based on the corresponding text, using the real image, corresponding text indicating the content of the real image, and a noise image. The information processing device 100 learns model M1 using an inversion method. This makes it possible for the information processing device 100 to generate (reproduce) the real image using model M1, which is a diffusion model (generative model).

[0102] In this case, the information processing device 100 uses the model M1, which generates (reproduces) a real image, and the text to generate mask information for the target range indicated by the text. For example, the information processing device 100 generates a target image (hereinafter also referred to as "image P30") which is a real image corresponding to the text using the model M1, and generates mask information for the target range indicated by the text within image P30 based on the attention information used by the model M1 when generating image P30.

[0103] In this way, the information processing device 100 generates mask information corresponding to real images, making it easy to add real image data to the training data for detection and segmentation tasks. Therefore, the information processing device 100 can appropriately generate other information using the information generated when the image is generated using the image generation model. The information processing device 100 may also learn a model M1 that generates real images using any learning method. For example, the information processing device 100 may learn a model M1 that generates real images based on a learning method related to diffusion models disclosed in Non-Patent Document 4, etc.

[0104] As described above, the information processing device 100 receives an image and text indicating the image, and is a model that has been trained to reconstruct the received image step by step from the noisy image and text, and the model is trained to reconstruct the image while taking into account attention based on the text. Furthermore, the information processing device 100 generates mask information for the range indicated by the text based on the attention used by the model when generating the image from the text. For example, the information processing device 100 generates mask information by performing inversion training. As a result, the information processing device 100 can generate mask information corresponding to specified character information.

[0105] Furthermore, the information processing device 100 may perform various processes in addition to generating mask information. For example, the information processing device 100 may perform processing on the generated image (target image) based on the mask information. The information processing device 100 may generate a processed image to be used as training data for machine learning by processing the target image. For example, the information processing device 100 may generate a processed image with the area of ​​the target image other than the target area identified based on the mask information as the background.

[0106] For example, the information processing device 100 generates mask information that distinguishes between the area of ​​the target image (image P30) that includes the object indicated by text TX2 (target area) and the area outside of it (excluded area), based on the attention information generated during the target image generation process in model M1, which takes text TX2 as input.

[0107] For example, the information processing device 100 identifies the area outside the target image (image P30) indicated by the generated mask information as the background. Then, the information processing device 100 generates a processed image by performing a predetermined processing operation on the area outside the target image (image P30) as the background. For example, the information processing device 100 generates image P31 as the processed image, which is the target image (image P30) with the area outside the target removed as the background.

[0108] Thus, the information processing device 100 can also be applied to tasks such as background removal of real images. The information processing device 100 may dynamically change (determine) which part of the image to use as the background, depending on the category in which the image is used (for example, the service field).

[0109] This point will be explained using the example of image P20 shown in Figure 2. For example, if the category is the interior category, the information processing device 100 identifies the portion of image P20 that is the target area, indicated by token TK1 as the "sofa," and the rest as the exclusion area. The information processing device 100 then removes the area of ​​image P20 that is not the target area (exclusion area) indicated by token TK1 as the background. As a result, the information processing device 100 generates image P32 as a processed image, which is image P20 with the area of ​​image P20 that is not the target area (exclusion area) indicated by token TK1 removed as the background.

[0110] Furthermore, for example, if the category is the fashion category, the information processing device 100 identifies the portion of image P20 that is the target area, which is the "woman wearing a dress" indicated by token TK2, and excludes the rest of the image. The information processing device 100 then removes the area of ​​image P20 that is not the target area (excluded area) indicated by token TK2 as the background. As a result, the information processing device 100 generates image P33 as a processed image, which is image P20 with the area of ​​image P20 that is not the target area (excluded area) indicated by token TK2 removed as the background.

[0111] [1-3-2. Examples of generating other information] Furthermore, the information processing device 100 may generate various types of feature information, such as feature vectors, in addition to mask information. For example, the information processing device 100 may use a diffusion model and input text to generate a corresponding image that corresponds to the input text, and generate feature information that indicates the features of the target range indicated by the input text.

[0112] In this case, the information processing device 100 may use the diffusion model and the input text to generate a corresponding image that corresponds to the input text, and based on the attention used by the diffusion model when generating the corresponding image, generate feature information that indicates the features of the target range indicated by the input text from the corresponding image. A third generation example in which the information processing device 100 generates feature information other than mask information using attention information A will be described below. Note that explanations of points similar to those described above will be omitted as appropriate.

[0113] In Figure 2, the information processing device 100 uses model M1 and text TX1 to generate image P20, which is a corresponding image to text TX1. Based on the attention information A used by model M1 when generating image P20, it generates feature information that indicates the features of the target range indicated by text TX1 within image P20. For example, the information processing device 100 generates a feature vector as feature information, which is obtained by extracting the features of the target indicated by text TX1 contained in image P20.

[0114] For example, the information processing device 100 generates feature information to be used as training data for machine learning. For example, the information processing device 100 may generate a feature vector that indicates the features of the object indicated by the text TX1 contained in the image P20 as ground truth information for a feature extraction model that extracts features from an image. For example, the information processing device 100 generates a feature vector that indicates the features of the object indicated by the text TX1 as ground truth information that the feature extraction model should output when an image containing the object indicated by the text TX1 is input.

[0115] For example, the information processing device 100 uses a feature extraction model M2, which is a feature extraction model that extracts features from an input image, to generate a feature vector that represents the features of the object indicated by the text. The feature extraction model M2 is a model that outputs feature information of the object indicated by the text in the image. Note that the feature extraction model that extracts features from an image can be any model that can extract features from an image, and a detailed explanation is omitted. For example, if a device other than the information processing device 100 (a learning device) is trained for model M2, the information processing device 100 may acquire model M2 by receiving it from the learning device. Note that the information processing device 100 may also train model M2.

[0116] For example, the information processing device 100 may use a processed image generated using attention information A as the image to be extracted, and generate a feature vector that indicates the characteristics of the object indicated by the text in the image to be extracted. For example, the information processing device 100 may use a processed image that has undergone background removal processing using attention information A as the image to be extracted, and generate a feature vector that indicates the characteristics of the object indicated by the text in the image to be extracted.

[0117] The information processing device 100 generates feature information of the object indicated by the text in the target image using the target image based on attention information A and the feature extraction model M2. The information processing device 100 inputs the target image based on attention information A into the model M2 and causes the model M2 to output a feature vector indicating the features of the object indicated by the text in the target image. In this way, the information processing device 100 generates a feature vector indicating the features of the object indicated by the text in the target image.

[0118] The information processing device 100 generates a feature vector representing the features of the target indicated by text TX2 in image P31 by inputting the background-removed image P31, which is generated using attention information A corresponding to text TX2, as the image to be extracted into model M2. As a result, the information processing device 100 generates a feature vector representing the features of the target range indicated by text TX2, "A woman wearing a dress is sitting on a sofa."

[0119] The information processing device 100 generates a feature vector that represents the features of the range (target range) in image P32 that includes the target "sofa" indicated by token TK1, by inputting the background-removed image P32, which is generated using attention information A1 corresponding to token TK1, into model M2 as the image to be extracted. In this way, the information processing device 100 generates a feature vector that represents the features of the target range corresponding to the target "sofa" indicated by token TK1.

[0120] The information processing device 100 generates a feature vector that represents the features of the range (target range) in image P33 that includes the target "woman wearing a dress" indicated by token TK2, by inputting the background-removed image P33, which is generated using attention information A2 corresponding to token TK2, into model M2 as the image to be extracted. In this way, the information processing device 100 generates a feature vector that represents the features of the target range corresponding to the target "woman wearing a dress" indicated by token TK2.

[0121] In this way, the information processing device 100 can appropriately generate other information using the information generated during image generation using an image generation model. As described above, the information processing device 100 generates a corresponding image indicated by the text using a diffusion model, and generates mask information for the target range indicated by the text based on the attention information used by the diffusion model when generating the corresponding image. Then, the information processing device 100 generates training data for learning the features of the image of the target indicated by the text by applying the mask corresponding to the text to the corresponding image. For example, as in the example above, the information processing device 100 generates a feature vector as feature information.

[0122] The above-described procedure is merely an example, and the information processing device 100 may generate various feature information using various information as appropriate. For example, instead of the Global Average Pooling process in the model, the information processing device 100 may generate information by processing using mask information. In this case, the information processing device 100 may generate a feature vector by performing pooling that emphasizes that part using the obtained mask information. For example, if the category is the interior category, the information processing device 100 generates a feature vector using the mask information corresponding to the object "sofa" indicated by token TK1 in image P20. For example, if the category is the fashion category, the information processing device 100 generates a feature vector using the mask information corresponding to the object "woman wearing a dress" indicated by token TK2 in image P20.

[0123] Furthermore, the information processing device 100 provides services using the generated feature information. For example, the information processing device 100 transmits the generated feature information to the terminal device 10 used by the user. For example, the information processing device 100 transmits to the terminal device 10 used by the user information that associates the feature information generated using the information generated during the image generation process in Model M1 with the generated image generated in Model M1.

[0124] For example, the information processing device 100 transmits to the user's terminal device 10 information that associates the generated image produced by model M1 with feature information generated using information from the image generation process and a label indicating the target corresponding to that feature information. For example, the information processing device 100 transmits to the user's terminal device 10 information that associates the generated image produced by model M1 with feature information and a label indicating the target corresponding to that feature information as training data.

[0125] [2. Configuration of the Information Processing System] Next, an example of the configuration of the information processing system 1 will be described using Figure 3. Figure 3 is a diagram showing an example of the configuration of an information processing system according to an embodiment. As shown in Figure 3, the information processing system 1 includes an information processing device 100, an information providing device 50, and a terminal device 10. The information processing device 100, the information providing device 50, and the terminal device 10 are connected to each other via a predetermined network N, either by wire or wireless communication. Note that the information processing system 1 shown in Figure 3 may include multiple information processing devices 100, multiple information providing devices 50, and multiple terminal devices 10.

[0126] The information processing device 100 is a computer that generates information using a model such as a diffusion model. In other words, the information processing device 100 is a generation device that performs information generation processing. The information processing device 100 uses a diffusion model and text to generate corresponding images corresponding to the text, and based on the attention used by the diffusion model when generating the corresponding images, it generates mask information for the target range indicated by the text within the corresponding images.

[0127] Furthermore, the information processing device 100 uses the corresponding text, which is text associated with the generated image that satisfies predetermined conditions, and the diffusion model to generate mask information for the target range indicated by the corresponding text within the generated image, based on the attention used by the diffusion model when generating the generated image. The information processing device 100 uses the diffusion model and the text to generate a corresponding image that corresponds to the text and generates feature information that indicates the features of the target range indicated by the text.

[0128] The information processing device 100 is a computer that provides services to users utilizing the terminal device 10. The information processing device 100 provides services using the generated mask information. The information processing device 100 provides the generated mask information to the user. The information processing device 100 transmits the generated mask information to the terminal device used by the user.

[0129] The information processing device 100 provides services using the generated feature information. The information processing device 100 provides the generated feature information to the user. The information processing device 100 transmits the generated feature information to the terminal device 10 used by the user.

[0130] The information provider 50 is a server device that provides information (data) usable for model training to the information processing device 100. For example, the information provider 50 transmits data that is permitted for use in model training to the information processing device 100 as training data. For example, the information provider 50 publishes public data on the internet and transmits the requested data to the requesting external device in response to a request from an external device such as the information processing device 100. For example, the information provider 50 receives request information from the information processing device 100 requesting data, and in response to the receipt of the request information, transmits the data corresponding to the request information to the information processing device 100.

[0131] For example, the information providing device 50 transmits to the information processing device 100, as training data, information that associates a group of images in which noise is gradually added to a single image with text that describes the object contained in that single image. For example, the information providing device 50 transmits to the information processing device 100, as training data, data that associates multiple step-by-step images in which noise is gradually added to an image of an object traded on an e-commerce site, etc., with text that describes the content of that object.

[0132] Terminal device 10 is an information processing device used by the user. Terminal device 10 can be implemented as, for example, a smartphone, a tablet device, a notebook PC (Personal Computer), a desktop PC, a mobile phone, or a PDA (Personal Digital Assistant). Note that terminal device 10 may sometimes be referred to as the user. In other words, the user can be read as terminal device 10.

[0133] The terminal device 10 accepts user operations. The terminal device 10 also transmits various information to the information processing device 100 and the information providing device 50 in response to user operations. For example, the terminal device 10 transmits request information to the information processing device 100 and the information providing device 50 to request various types of information. The terminal device 10 also receives various types of information from the information processing device 100 and the information providing device 50.

[0134] Terminal device 10 transmits the image to be used for information generation (target image) specified by the user to the information processing device 100. Terminal device 10 transmits information indicating the target specified by the user (specified information) to the information processing device 100.

[0135] The terminal device 10 receives various types of information from the information processing device 100. For example, the terminal device 10 receives mask information from the information processing device 100. For example, the terminal device 10 receives feature information from the information processing device 100.

[0136] Terminal device 10 receives information from information processing device 100 as training data, which associates the generated image produced by model M1 with mask information and a label indicating the target corresponding to that mask information. Terminal device 10 receives information from information processing device 100 as training data, which associates the generated image produced by model M1 with feature information and a label indicating the target corresponding to that feature information.

[0137] [3. Configuration of the Information Processing Device] Next, the configuration of the information processing device 100 according to the embodiment will be described using Figure 4. Figure 4 is a diagram showing an example of the configuration of the information processing device 100 according to the embodiment. As shown in Figure 4, the information processing device 100 has a communication unit 110, a storage unit 120, and a control unit 130. The information processing device 100 may also have an input unit (for example, a keyboard or mouse) that receives various operations from an administrator of the information processing device 100, and a display unit (for example, a liquid crystal display) for displaying various information.

[0138] (Communications Department 110) The communication unit 110 is implemented, for example, by a NIC (Network Interface Card). The communication unit 110 is connected to the network by wire or wireless connection and transmits and receives information with the terminal device 10 and the information providing device 50.

[0139] (Storage unit 120) The storage unit 120 is implemented by, for example, semiconductor memory elements such as RAM (Random Access Memory) and flash memory, or by storage devices such as hard disks and optical discs. As shown in Figure 4, the storage unit 120 according to this embodiment has a learning data storage unit 121 and a model information storage unit 122.

[0140] (Learning data storage unit 121) The learning data storage unit 121 according to this embodiment stores various information (learning data) related to the data used for learning. The learning data storage unit 121 stores the training data used to generate the model. The learning data storage unit 121 stores learning data information, including information such as the data used for input and the correct answer information (correct answer image) corresponding to that data.

[0141] The learning data storage unit 121 stores learning data in which combinations of noise images and text used as input are associated with image data (correct images) corresponding to that input.

[0142] The learning data storage unit 121 is not limited to the above and may store various types of information depending on the purpose. For example, the learning data storage unit 121 may store information indicating the type of input data, associated with each input data.

[0143] (Model information storage unit 122) The model information storage unit 122 according to the embodiment stores information about the model. For example, the model information storage unit 122 stores information (model data) of a trained model (model) that has been trained (generated) through a training process. Figure 5 is a diagram showing an example of the model information storage unit according to the embodiment. In the example shown in Figure 5, the model information storage unit 122 includes items such as "model ID", "purpose", and "model data".

[0144] The "Model ID" indicates identification information for identifying the model. The "Purpose" indicates the purpose of the corresponding model. The "Model Data" indicates the data of the model. Figure 5 shows an example where conceptual information such as "MDT1" is stored in "Model Data," but in reality, it includes various information that constitutes the model, such as information about the model's configuration (network configuration) and parameters. For example, "Model Data" includes information such as the nodes in each layer of the network, the functions adopted by each node, the connection relationships between nodes, and the connection coefficients set for the connections between nodes.

[0145] Figure 5 shows that the model identified by model ID "M1" (model M1) is used for "image generation," for example, model M1 is used to generate images that represent the content of text. For example, model M1 is used to convert text information into image information, that is, to represent (reproduce) the content of text information as an image. It also shows that the model data for model M1 is model data MDT1.

[0146] Furthermore, the model information storage unit 122 may store various types of information depending on the purpose, not limited to those described above. Also, models such as Model M1 may be intended for use as program modules that are part of artificial intelligence software. Models such as Model M1 may also be programs.

[0147] (Control unit 130) Returning to the explanation of Figure 4, the control unit 130 is a controller, and is realized by various programs (corresponding to examples of information processing programs such as learning programs and generation programs) stored in the memory device inside the information processing device 100 being executed using RAM as the working area by a CPU (Central Processing Unit) or MPU (Micro Processing Unit), etc. Alternatively, the control unit 130 is a controller and can be realized by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array), for example.

[0148] As shown in Figure 4, the control unit 130 includes an acquisition unit 131, a generation unit 132, a learning unit 133, a processing unit 134, and a providing unit 135, and realizes or executes the information processing functions and operations described below. Note that the internal configuration of the control unit 130 is not limited to the configuration shown in Figure 4, and other configurations are also acceptable as long as they perform the information processing described later.

[0149] (Acquisition part 131) The acquisition unit 131 acquires various types of information. The acquisition unit 131 acquires various types of information from external information processing devices such as the terminal device 10 and the information providing device 50. The acquisition unit 131 may also receive various types of information, such as those shown in the learning data storage unit 121 and the model information storage unit 122, from external information processing devices.

[0150] The acquisition unit 131 acquires various information from the storage unit 120. The acquisition unit 131 acquires various information from the learning data storage unit 121, the model information storage unit 122, etc. The acquisition unit 131 receives various information from external devices.

[0151] The acquisition unit 131 receives various information from the terminal device 10 used by the user. For example, the acquisition unit 131 receives an image from the terminal device 10 that is to be used to generate mask information. For example, the acquisition unit 131 receives information from the terminal device 10 that specifies the object from the image to be used to generate mask information. For example, the acquisition unit 131 receives an image from the terminal device 10 that is to be used to generate feature information.

[0152] The acquisition unit 131 receives various types of information from the information providing device 50. For example, the acquisition unit 131 receives information used for learning from the information providing device 50. For example, the acquisition unit 131 receives training data used for training model M1 from the information providing device 50.

[0153] The acquisition unit 131 acquires a diffusion model, which is a model that has been trained to progressively reconstruct the image indicated by the text from a noisy image, and is a model that has been trained to reconstruct the image according to attention based on the text. The acquisition unit 131 acquires the input text entered by the user. The acquisition unit 131 acquires specification information that specifies one of the tokens contained in the input text. The acquisition unit 131 acquires specification information that specifies multiple tokens contained in the input text.

[0154] The acquisition unit 131 acquires a target image, which is an image that satisfies predetermined conditions, and corresponding text, which is text associated with the target image. The acquisition unit 131 acquires a diffusion model, which is a model that has been trained to reconstruct the target image step by step from the noise image and the corresponding text, and which reconstructs the target image according to attention based on the corresponding text.

[0155] The acquisition unit 131 acquires a target image, which is a real image, and corresponding text associated with the target image. The acquisition unit 131 acquires a target image, which is an image taken of a predetermined object, and corresponding text indicating the predetermined object. The acquisition unit 131 acquires corresponding text indicating the object captured by the target image. The acquisition unit 131 acquires corresponding text indicating the object included in the target image.

[0156] The acquisition unit 131 acquires the target image to be processed from the user. The acquisition unit 131 also acquires specification information from the user that specifies the target for information generation within the image.

[0157] (Generation unit 132) The generation unit 132 performs generation processing to estimate various types of information. For example, the generation unit 132 performs generation processing based on various types of information acquired by the acquisition unit 131. The generation unit 132 performs generation processing based on various types of information stored in the storage unit 120. The generation unit 132 performs generation processing based on various types of information received from an external information processing device. The generation unit 132 performs generation processing based on a model learned by the learning unit 133. The generation unit 132 performs generation processing based on information determined by the processing unit 134.

[0158] The generation unit 132 generates a corresponding image for the input text using the diffusion model acquired by the acquisition unit 131 and the input text, and generates mask information for the target range indicated by the input text within the corresponding image based on the attention used by the diffusion model when generating the corresponding image. The generation unit 132 generates a corresponding image for the input text, and generates mask information for the target range indicated by the tokens within the corresponding image based on the attention used by the diffusion model when generating the corresponding image for each token contained in the input text.

[0159] The generation unit 132 generates mask information for the target range indicated by a single token in the corresponding image, based on the attention corresponding to a single token specified by the specified information among the tokens contained in the input text. The generation unit 132 generates mask information for the target range indicated by multiple tokens in the corresponding image, based on the attention corresponding to each of multiple tokens specified by the specified information among the tokens contained in the input text.

[0160] The generation unit 132 generates mask information for the target range corresponding to the logical OR of multiple tokens. The generation unit 132 generates mask information for the target range corresponding to the logical AND of multiple tokens.

[0161] The generation unit 132 generates mask information for the target range indicated by each token in the corresponding image based on the attention corresponding to each noun token in the input text. The generation unit 132 generates mask information for the target range indicated by each token in the corresponding image based on the attention corresponding to each verb token in the input text. The generation unit 132 generates a corresponding image corresponding to the input text and generates mask information for the target range indicated by the input text in the corresponding image based on the attention at a predetermined stage during the diffusion model generation process.

[0162] The generation unit 132 uses the diffusion model and corresponding text acquired by the acquisition unit 131 to generate mask information for the target range indicated by the corresponding text within the target image, based on the attention used by the diffusion model when generating the target image. The generation unit 132 generates mask information for the target range corresponding to a predetermined target within the target image, based on the attention used by the diffusion model when generating the target image.

[0163] The generation unit 132 generates mask information for the target range in the target image that corresponds to the object being captured, based on the attention used by the diffusion model when generating the target image. The generation unit 132 generates a corresponding image that corresponds to the corresponding text, and generates mask information for the target range indicated by the corresponding text in the corresponding image, based on the attention at a predetermined stage during the diffusion model generation process.

[0164] The generation unit 132 uses the diffusion model acquired by the acquisition unit 131 and the input text to generate a corresponding image that corresponds to the input text, and generates feature information that indicates the characteristics of the target range indicated by the input text. The generation unit 132 generates a vector that indicates the characteristics of the target range indicated by the input text as feature information. The generation unit 132 generates feature information to be used as training data for machine learning.

[0165] The generation unit 132 generates a corresponding image corresponding to the input text, and based on the attention used by the diffusion model when generating the corresponding image, generates feature information from the corresponding image that indicates the features of the target range indicated by the input text. The generation unit 132 generates a corresponding image corresponding to the input text, and based on the attention used by the diffusion model when generating the corresponding image that corresponds to each token contained in the input text, generates feature information from the corresponding image that indicates the features of the target range indicated by the token.

[0166] The generation unit 132 generates feature information that indicates the characteristics of the target range shown by a single token in the corresponding image, based on the attention corresponding to a single token specified by the specified information among the tokens included in the input text. The generation unit 132 generates feature information that indicates the characteristics of the target range shown by multiple tokens in the corresponding image, based on the attention corresponding to each of multiple tokens specified by the specified information among the tokens included in the input text.

[0167] The generation unit 132 generates feature information that indicates the characteristics of the target range corresponding to the logical OR of multiple tokens. The generation unit 132 generates feature information that indicates the characteristics of the target range corresponding to the logical AND of multiple tokens.

[0168] The generation unit 132 generates feature information from the corresponding image that indicates the range of the target area indicated by each noun token in the input text, based on the attention corresponding to each noun token in the input text. The generation unit 132 generates feature information from the corresponding image that indicates the range of the target area indicated by each verb token in the input text, based on the attention corresponding to each verb token in the input text. The generation unit 132 generates a corresponding image corresponding to the input text and generates feature information from the corresponding image that indicates the range of the target area indicated by the input text, based on the attention at a predetermined stage during the diffusion model generation process.

[0169] (Learning Section 133) The learning unit 133 learns various information. The learning unit 133 generates various models through learning processing. For example, the learning unit 133 performs learning processing based on information stored in the learning data storage unit 121, etc. The learning unit 133 generates various models based on information stored in the learning data storage unit 121, etc. For example, the learning unit 133 learns model M1 using the learning data stored in the learning data storage unit 121.

[0170] The learning unit 133 is a model that has been trained to gradually reconstruct the image indicated by the text from a noisy image, and it learns a diffusion model that has been trained to reconstruct the image in accordance with attention based on the text. The learning unit 133 is a model that has been trained to gradually reconstruct the target image from a noisy image and the corresponding text, and it learns a diffusion model that has been trained to reconstruct the target image in accordance with attention based on the corresponding text.

[0171] The learning unit 133 learns a diffusion model using multiple stage images acquired by the acquisition unit 131. The learning unit 133 learns the diffusion model to generate a stage image which is an image obtained by removing one level of noise from one of the multiple stage images. The learning unit 133 learns the diffusion model to generate a first stage image which is obtained by removing one level of noise from the second stage image. The learning unit 133 learns the diffusion model to generate a second stage image which is obtained by removing one level of noise from the third stage image.

[0172] For example, the learning unit 133 performs learning processing based on various information acquired by the acquisition unit 131. For example, the learning unit 133 learns a model based on information generated by the generation unit 132. The learning unit 133 performs learning processing based on information from an external information processing device and information stored in the storage unit 120. The learning unit 133 performs learning processing based on information stored in the model information storage unit 122. The learning unit 133 stores the model generated by learning in the model information storage unit 122.

[0173] The learning unit 133 performs learning processing. The learning unit 133 performs various types of learning. The learning unit 133 learns various types of information based on the information acquired by the acquisition unit 131. The learning unit 133 learns (generates) a model. The learning unit 133 learns various types of information such as the model. The learning unit 133 generates a model through learning. The learning unit 133 learns the model using various machine learning techniques. For example, the learning unit 133 learns the parameters of the model (network). The learning unit 133 learns the model using various machine learning techniques.

[0174] The learning unit 133 generates various learning models such as model M1. The learning unit 133 learns the network parameters. For example, the learning unit 133 learns the network parameters of various learning models such as model M1. The learning unit 133 generates various learning models such as model M1 by performing learning processing using the learning data stored in the model information storage unit 122. For example, the learning unit 133 generates a model used for image generation. The learning unit 133 generates various learning models such as model M1 by learning the network parameters of various learning models such as model M1.

[0175] The learning unit 133 performs learning processing based on the training data (teacher data) stored in the model information storage unit 122. The learning unit 133 generates various learning models, such as model M1, by performing learning processing using the training data stored in the model information storage unit 122.

[0176] For example, the learning unit 133 performs learning processing using methods such as backpropagation so that the image output by model M1 approaches the correct information (information indicating the desired generation result) associated with the input information input to model M1. For example, the learning unit 133 performs learning processing so that the image output by model M1, which has input information including a noisy image, approaches the correct information (image from which the noise in the input noisy image has been removed by one level).

[0177] For example, the learning unit 133 adjusts the values ​​of the weights (i.e., connection coefficients) that are considered when values ​​are transmitted between nodes during the learning process. In this way, the learning unit 133 learns model M1 by processing such as backpropagation to correct the parameters (connection coefficients) so that the error between the output of model M1 and the correct information corresponding to the input is reduced. For example, the learning unit 133 generates model M1 by processing such as backpropagation to minimize a predetermined loss function. This allows the learning unit 133 to perform a learning process to learn the parameters of model M1.

[0178] The model training method is not limited to the methods described above, and any known technique can be applied. Furthermore, the generation of each model may be performed using various conventional machine learning techniques as appropriate. For example, the model may be generated using supervised machine learning techniques such as SVM (Support Vector Machine). Alternatively, the model may be generated using unsupervised machine learning techniques. For example, the model may be generated using deep learning techniques. For example, the model may be generated using various deep learning techniques such as DNN (Deep Neural Network), RNN (Recurrent Neural Network), and CNN (Convolutional Neural Network) as appropriate. The above description of model generation is illustrative, and the model may be generated using a training method appropriately selected according to the available information. In other words, the training unit 133 may generate model M1 by any method as long as it can train model M1 to output information corresponding to the correct answer information when input information included in the training data is input.

[0179] As described above, the learning method used by the learning unit 133 is not particularly limited, but for example, training data may be prepared by linking data (input information) with correct answer information (output information), and this training data may be input into a computational model based on a multilayer neural network for training. Alternatively, methods based on DNNs such as CNNs and 3D-CNNs may be used. Specifically, the learning method used by the learning unit 133 may be a method based on U-nets. When the learning unit 133 deals with time-series data such as audio, it may use methods based on recurrent neural networks (RNNs) or LSTMs (Long Short-Term Memory units), which are extensions of RNNs.

[0180] (Processing Unit 134) The processing unit 134 processes various types of information. The processing unit 134 performs information processing based on the information stored in the memory unit 120. The processing unit 134 functions as an inference unit that performs inference processing. The processing unit 134 performs inference processing using the model stored in the model information memory unit 122. For example, the learning unit 133 performs inference processing based on the information generated by the generation unit 132.

[0181] The processing unit 134 performs processing on the target image based on the mask information. The processing unit 134 generates a processed image to be used as training data for machine learning through the processing of the target image. The processing unit 134 generates a processed image with the area of ​​the target image other than the target area identified based on the mask information as the background.

[0182] The processing unit 134 performs inference using the model M1 learned by the learning unit 133. The processing unit 134 performs inference using the diffusion model model M1. The processing unit 134 inputs a noise image and text as input information to the model M1 and performs inference processing to generate an image containing the object corresponding to the text.

[0183] (Provider 135) The providing unit 135 provides various types of information. For example, the providing unit 135 provides various types of information to the terminal device 10 and the information providing device 50. The providing unit 135 transmits various types of information to the terminal device 10 and the information providing device 50. The providing unit 135 provides various types of information based on the various types of information acquired by the acquisition unit 131. The providing unit 135 provides various types of information based on the various types of information generated by the generation unit 132.

[0184] The providing unit 135 provides various information based on the various information learned by the learning unit 133. The providing unit 135 provides various information based on the model learned by the learning unit 133. The providing unit 135 provides various information based on the various information processed by the processing unit 134. For example, the providing unit 135 transmits information to the terminal device 10.

[0185] The service provider 135 provides services using the diffusion model learned by the learning unit 133. The service provider 135 provides services using the mask information generated by the generation unit 132. The service provider 135 provides the mask information generated by the generation unit 132 to the user. The service provider 135 transmits the mask information generated by the generation unit 132 to the terminal device used by the user.

[0186] The provisioning unit 135 provides services using the feature information generated by the generation unit 132. The provisioning unit 135 provides the feature information generated by the generation unit 132 to the user. The provisioning unit 135 transmits the feature information generated by the generation unit 132 to the terminal device 10 used by the user.

[0187] [4. Information Processing Flow] Next, the information processing flow by the information processing device 100 will be explained using Figures 6 to 8. Figures 6 to 8 are flowcharts showing an example of information processing according to the embodiment.

[0188] First, let's explain the processing example shown in Figure 6. As shown in Figure 6, the information processing device 100 acquires a diffusion model, which is a model that has been trained to gradually reconstruct the image indicated by the text from a noisy image and has been trained to reconstruct the image in response to attention based on the text, and the input text entered by the user (step S101).

[0189] The information processing device 100 uses a diffusion model and input text to generate a corresponding image corresponding to the input text, and the diffusion model generates mask information that masks the target range indicated by the input text within the corresponding image based on the attention used when generating the corresponding image (step S102).

[0190] Next, an example of processing shown in Figure 7 will be described. As shown in Figure 7, the information processing device 100 acquires a target image, which is an image that satisfies predetermined conditions, a corresponding text, which is text associated with the target image, and a diffusion model, which is a model that has been trained to reconstruct the target image step by step from the noise image and the corresponding text, and which reconstructs the target image in accordance with attention based on the corresponding text (step S201).

[0191] The information processing device 100 uses the diffusion model and the corresponding text to generate mask information that masks the target range indicated by the corresponding text within the target image, based on the attention used when the diffusion model generated the target image (step S202).

[0192] Next, we will explain the processing example shown in Figure 8. As shown in Figure 8, the information processing device 100 acquires a diffusion model, which is a model that has been trained to gradually reconstruct the image indicated by the text from a noisy image and has been trained to reconstruct the image in response to attention based on the text, and the input text entered by the user (step S301).

[0193] The information processing device 100 uses a diffusion model and the input text to generate a corresponding image that corresponds to the input text, and generates feature information that indicates the characteristics of the target range indicated by the input text (step S302).

[0194] [5. Effects] As described above, the information processing device 100 according to the embodiment includes an acquisition unit 131 and a generation unit 132. The acquisition unit 131 acquires a target image, which is an image that satisfies predetermined conditions, a corresponding text, which is text associated with the target image, and a diffusion model, which is a model that has been trained to gradually reconstruct the target image from a noise image and the corresponding text, and which reconstructs the target image in accordance with attention based on the corresponding text. The generation unit 132 uses the diffusion model and corresponding text acquired by the acquisition unit 131 to generate mask information for the target range indicated by the corresponding text within the target image, based on the attention used by the diffusion model when generating the target image.

[0195] Thus, the information processing device 100 according to this embodiment can generate a target image corresponding to the corresponding text using a diffusion model trained to reconstruct an image in response to text-based attention and the corresponding text, and can generate mask information for the target range indicated by the corresponding text within the target image based on the attention used by the diffusion model when generating the target image. Therefore, the information processing device 100 can appropriately generate other information using the information obtained when generating an image using the image generation model.

[0196] Furthermore, in the information processing device 100 according to this embodiment, the acquisition unit 131 acquires a target image which is a real image and corresponding text associated with the target image.

[0197] Thus, the information processing device 100 according to this embodiment can generate mask information for a real image. Therefore, the information processing device 100 can appropriately generate other information using the information obtained during image generation using the image generation model.

[0198] Furthermore, in the information processing device 100 according to the embodiment, the acquisition unit 131 acquires an image of a predetermined object, which is an image of a predetermined object, and corresponding text indicating the predetermined object. The generation unit 132 generates mask information for the target range corresponding to the predetermined object within the image, based on the attention used by the diffusion model when generating the image of the predetermined object.

[0199] Thus, the information processing device 100 according to this embodiment can generate mask information using a target image, which is an image captured of a predetermined object. Therefore, the information processing device 100 can appropriately generate other information using the information obtained during image generation using an image generation model.

[0200] Furthermore, in the information processing device 100 according to the embodiment, the acquisition unit 131 acquires corresponding text indicating the object being captured by the target image. The generation unit 132 generates mask information for the target range in the target image that corresponds to the object being captured, based on the attention used by the diffusion model when generating the target image.

[0201] Thus, the information processing device 100 according to this embodiment can generate mask information corresponding to the corresponding text indicating the imaged object captured by the target image. Therefore, the information processing device 100 can appropriately generate other information using the information obtained during image generation using the image generation model.

[0202] Furthermore, in the information processing device 100 according to the embodiment, the acquisition unit 131 acquires corresponding text indicating the imaged object included in the target image.

[0203] Thus, the information processing device 100 according to this embodiment can generate mask information corresponding to the corresponding text indicating the imaged object included in the target image. Therefore, the information processing device 100 can appropriately generate other information using the information obtained during image generation using the image generation model.

[0204] Furthermore, the information processing device 100 according to this embodiment includes a processing unit 134. The processing unit 134 performs processing on the target image based on the mask information.

[0205] Thus, the information processing device 100 according to this embodiment can appropriately process an image by performing processing on the target image based on the mask information.

[0206] Furthermore, in the information processing device 100 according to the embodiment, the processing unit 134 generates a processed image to be used as training data for machine learning by processing the target image.

[0207] Thus, the information processing device 100 according to this embodiment can appropriately generate training data by performing processing on the target image based on the mask information.

[0208] Furthermore, in the information processing device 100 according to this embodiment, the processing unit 134 generates a processed image with the area of ​​the target image other than the target area identified based on the mask information as the background.

[0209] Thus, the information processing device 100 according to this embodiment can appropriately generate training data by generating a processed image with a background that includes an area other than the target area identified based on the mask information.

[0210] Furthermore, in the information processing device 100 according to the embodiment, the generation unit 132 generates a corresponding image corresponding to the corresponding text, and generates mask information for the target range indicated by the corresponding text from the corresponding image based on attention at a predetermined stage during the diffusion model generation process.

[0211] Thus, the information processing device 100 according to this embodiment generates a corresponding image corresponding to the corresponding text, and based on attention at a predetermined stage during the diffusion model generation process, generates mask information for the target range indicated by the corresponding text within the corresponding image. This allows for the appropriate generation of other information using the information obtained during image generation using the image generation model.

[0212] Furthermore, the information processing device 100 according to the embodiment includes a learning unit 133. The learning unit 133 is a diffusion model, which is a model that has been trained to reconstruct a target image step by step from a noise image and corresponding text, and learns to reconstruct the target image in accordance with attention based on the corresponding text.

[0213] Thus, the information processing device 100 according to this embodiment can learn a model that appropriately generates images by learning a diffusion model that has been trained to reconstruct images in response to attention based on text.

[0214] Furthermore, the information processing device 100 according to this embodiment has a providing unit 135. The providing unit 135 provides a service using the mask information generated by the generation unit 132.

[0215] Thus, the information processing device 100 according to this embodiment can provide appropriate services by providing services using the generated mask information.

[0216] Furthermore, in the information processing device 100 according to this embodiment, the providing unit 135 provides the user with the mask information generated by the generation unit 132.

[0217] Thus, the information processing device 100 according to this embodiment can provide appropriate services to the user by providing mask information to the user.

[0218] Furthermore, in the information processing device 100 according to this embodiment, the providing unit 135 transmits the mask information generated by the generation unit 132 to the terminal device 10 used by the user.

[0219] Thus, the information processing device 100 according to this embodiment can provide the user with appropriate search results by transmitting mask information to the terminal device 10 used by the user.

[0220] [6. Hardware Configuration] Furthermore, the information processing device 100 and terminal device 10 according to the above-described embodiment are realized by a computer 1000 having a configuration such as that shown in Figure 11. The following explanation will use the information processing device 100 as an example. Figure 11 is a diagram showing an example of the hardware configuration. The computer 1000 is connected to an output device 1010 and an input device 1020, and has a configuration in which an arithmetic unit 1030, a primary storage device 1040, a secondary storage device 1050, an output interface 1060, an input interface 1070, and a network interface 1080 are connected by a bus 1090.

[0221] The arithmetic unit 1030 operates based on programs stored in the primary storage device 1040 and the secondary storage device 1050, as well as programs read from the input device 1020, and executes various processes. The arithmetic unit 1030 can be implemented using, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), an ASIC (Application Specific Integrated Circuit), or an FPGA (Field Programmable Gate Array).

[0222] The primary storage device 1040 is a memory device, such as RAM (Random Access Memory), that temporarily stores data used by the arithmetic unit 1030 for various calculations. The secondary storage device 1050 is a storage device where data used by the arithmetic unit 1030 for various calculations and various databases are registered, and can be implemented using ROM (Read Only Memory), HDD (Hard Disk Drive), SSD (Solid State Drive), flash memory, etc. The secondary storage device 1050 may be internal storage or external storage. The secondary storage device 1050 may also be a removable storage medium such as a USB memory stick or an SD (Secure Digital) memory card. Furthermore, the secondary storage device 1050 may be cloud storage (online storage), NAS (Network Attached Storage), a file server, etc.

[0223] The output I / F 1060 is an interface for transmitting information to be output to output devices 1010, such as displays, projectors, and printers, and is implemented using connectors of standards such as USB (Universal Serial Bus), DVI (Digital Visual Interface), and HDMI (High Definition Multimedia Interface). The input I / F 1070 is an interface for receiving information from various input devices 1020, such as mice, keyboards, keypads, buttons, and scanners, and is implemented using, for example, USB.

[0224] Furthermore, the output interface 1060 and input interface 1070 may be wirelessly connected to the output device 1010 and input device 1020, respectively. In other words, the output device 1010 and input device 1020 may be wireless devices.

[0225] Furthermore, the output device 1010 and the input device 1020 may be integrated as a touch panel. In this case, the output I / F 1060 and the input I / F 1070 may also be integrated as an input / output I / F.

[0226] The input device 1020 may also be a device that reads information from, for example, an optical recording medium such as a CD (Compact Disc), DVD (Digital Versatile Disc), or PD (Phase Change Rewritable Disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory.

[0227] The network interface 1080 receives data from other devices via network N and sends it to the computing unit 1030, and also transmits data generated by the computing unit 1030 to other devices via network N.

[0228] The arithmetic unit 1030 controls the output device 1010 and the input device 1020 via the output interface 1060 and the input interface 1070. For example, the arithmetic unit 1030 loads a program from the input device 1020 or the secondary storage device 1050 onto the primary storage device 1040 and executes the loaded program.

[0229] For example, when computer 1000 functions as an information processing device 100, the arithmetic unit 1030 of computer 1000 realizes the functions of the control unit 130 by executing a program loaded onto the primary storage device 1040. Alternatively, the arithmetic unit 1030 of computer 1000 may load a program obtained from another device via the network interface 1080 onto the primary storage device 1040 and execute the loaded program. Furthermore, the arithmetic unit 1030 of computer 1000 may cooperate with other devices via the network interface 1080 and call and use program functions, data, etc., from other programs on other devices.

[0230] [7. Other] Furthermore, among the processes described in the above embodiments and modifications, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by known methods. In addition, the processing procedures, specific names, and information including various data and parameters shown in the above document and drawings can be changed at will unless otherwise specified. For example, the various information shown in each figure is not limited to the information shown.

[0231] Furthermore, the components of each illustrated device are functionally conceptual and do not necessarily need to be physically configured as shown. In other words, the specific forms of distribution and integration of each device are not limited to those shown, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads and usage conditions.

[0232] Furthermore, the embodiments and modifications described above can be combined as appropriate, provided that the processing content is not inconsistent.

[0233] Furthermore, the terms "section, module, unit" mentioned above can be replaced with "means" or "circuit," etc. For example, the acquisition unit can be replaced with acquisition means or acquisition circuit. [Explanation of symbols]

[0234] 1. Information Processing System 100 Information Processing Devices 121 Learning data storage unit 122 Model Information Storage Unit 130 Control Unit 131 Acquisition Department 132 Generation part 133 Learning Department 134 Processing Unit 135 Provision Department 10 Terminal devices 50 Information provision device N Network

Claims

1. An acquisition unit that acquires a target image which is an image that satisfies predetermined conditions, a corresponding text which is text associated with the target image, and a diffusion model which is a model that has been trained to reconstruct the target image step by step from a noise image and the corresponding text, and is a model that has been trained to reconstruct the target image in accordance with attention based on the corresponding text. A generation unit generates a processed image to be used as training data for machine learning by using the diffusion model and corresponding text acquired by the acquisition unit, and based on the attention used by the diffusion model when generating the target image, masking the target range indicated by the corresponding text within the target image, and performing processing on the target image based on the mask information. An information processing device characterized by comprising:

2. The acquisition unit is, Obtain the target image, which is a real image, and the corresponding text associated with the target image. The information processing apparatus according to feature 1.

3. The acquisition unit is, The target image, which is an image of a predetermined target, and the corresponding text indicating the predetermined target are obtained. The generating unit is Based on the attention used by the diffusion model when generating the target image, the diffusion model generates mask information that masks the target range corresponding to the predetermined target within the target image. The information processing apparatus according to feature 1.

4. The acquisition unit is, The corresponding text indicating the target image captured by the aforementioned target image is obtained, The generating unit is Based on the attention used by the diffusion model when generating the target image, the diffusion model generates mask information that masks the target range in the target image that corresponds to the object being captured. The information processing apparatus according to claim 3.

5. The acquisition unit is, The corresponding text indicating the target object to be captured, which is included in the target image, is acquired. The information processing apparatus according to feature 4.

6. A processing unit that performs processing on the target image based on the mask information, The information processing apparatus according to claim 1, characterized by comprising:

7. The aforementioned processing unit, By processing the aforementioned target image, a processed image is generated to be used as training data for machine learning. The information processing apparatus according to feature 6.

8. The aforementioned processing unit, A processed image is generated using the area of ​​the target image other than the target area identified based on the mask information as the background. The information processing apparatus according to feature 6.

9. The generating unit is A corresponding image is generated that corresponds to the corresponding text, and based on attention at a predetermined stage during the diffusion model generation process, mask information is generated that masks the target range indicated by the corresponding text within the corresponding image. The information processing apparatus according to feature 1.

10. A learning unit that learns the diffusion model, which is a model that has been trained to reconstruct the target image step by step from the noise image and the corresponding text, and which is a model that has been trained to reconstruct the target image in accordance with attention based on the corresponding text. The information processing apparatus according to claim 1, characterized by comprising:

11. A service provider that provides services using the mask information generated by the generation unit. The information processing apparatus according to claim 1, further comprising:

12. The aforementioned supply unit is, The mask information generated by the generation unit is provided to the user. The information processing apparatus according to feature 11.

13. The aforementioned supply unit is, The mask information generated by the generation unit is transmitted to the terminal device used by the user. The information processing apparatus according to feature 12.

14. A method of information processing performed by a computer, An acquisition step of acquiring a target image which is an image that satisfies predetermined conditions, a corresponding text which is text associated with the target image, and a diffusion model which is a model that has been trained to reconstruct the target image step by step from a noise image and the corresponding text, and is a model that has been trained to reconstruct the target image in accordance with attention based on the corresponding text. A generation step in which, using the diffusion model and corresponding text obtained in the acquisition step, the diffusion model generates mask information that masks the target range indicated by the corresponding text within the target image based on the attention used when generating the target image, and performs processing on the target image based on the mask information to generate a processed image to be used as training data for machine learning, An information processing method characterized by including

15. An acquisition procedure for acquiring a target image which is an image that satisfies predetermined conditions, a corresponding text which is text associated with the target image, and a diffusion model which is a model that has been trained to reconstruct the target image step by step from a noise image and the corresponding text, and which is a model that has been trained to reconstruct the target image in accordance with attention based on the corresponding text. A generation procedure for generating a processed image to be used as training data for machine learning, which involves using the diffusion model and corresponding text obtained by the acquisition procedure, generating mask information that masks the target range indicated by the corresponding text within the target image based on the attention used by the diffusion model when generating the target image, and performing processing on the target image based on the mask information; An information processing program characterized by causing a computer to execute it.

Images