Information processing device and information processing program

Abstract

To generate, from an image, a detailed image description sentence that describes each object included in the image.SOLUTION: An information processing device includes: an acquisition unit that acquires a visual language model which is a machine learning model for generating, from an image, an image description sentence that is a sentence for describing a content of the image; and a model generation unit that generates an attribute visual language model which is a visual language model for generating, from the image, an image description sentence including attribute information indicating an attribute of an object included in the image.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] The present invention relates to an information processing device and an information processing program. [Background technology]

[0002] Conventionally, there are known techniques for generating image captions (also called image captions, hereinafter referred to as "image descriptions") from images. For example, there is known a technique related to a visual language model (VLM), which is a machine learning model that generates image descriptions from images. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, Yonghui Wu, " CoCa: Contrastive Captioners are Image-Text Foundation Models ", [online], 14 Jun 2022, [Retrieved February 9, 2024], Internet<URL:https: / / arxiv.org / pdf / 2205.01917.pdf> Summary of the Invention [Problem to be solved by the invention]

[0004] However, the above-mentioned conventional technology only generates an image description from an image, and therefore does not necessarily enable the generation of a detailed image description that explains each of the objects contained in the image.

[0005] An object of the present application is to provide an information processing device and an information processing program that are capable of generating, from an image, a detailed image caption that explains each of the objects included in the image. [Means for solving the problem]

[0006] The information processing device according to the present application includes an acquisition unit that acquires a visual language model, which is a machine learning model that generates an image description, which is a sentence that explains the contents of an image, from an image, and a model generation unit that generates an attribute visual language model, which is the visual language model that generates the image description from the image, including attribute information that indicates the attributes of an object included in the image. [Effects of the Invention]

[0007] According to one aspect of the embodiment, it is possible to generate a detailed image description from an image that describes each of the objects contained in the image. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating a visual language model according to an embodiment. [Figure 2] FIG. 2 is a diagram showing an example of an image description generated by a visual language model according to the prior art. [Figure 3] FIG. 3 is a diagram showing an outline of a process for generating an image caption using a visual language model according to the prior art. [Figure 4] FIG. 4 is a diagram showing an outline of the process of generating an image description using a color visual language model according to the embodiment. [Figure 5] FIG. 5 is a diagram illustrating an example of an image description generated by the color visual language model according to the embodiment. [Figure 6] FIG. 6 is a diagram illustrating an example of the configuration of an information processing device according to the embodiment. [Figure 7] FIG. 7 is a diagram illustrating an overview of information processing by the information processing device according to the embodiment. [Figure 8]FIG. 8 is a flowchart showing the procedure of information processing by the information processing device according to the embodiment. [Figure 9] FIG. 9 is a diagram showing an outline of information processing by an information processing device according to a modified example. [Figure 10] FIG. 10 is a hardware configuration diagram illustrating an example of a computer that realizes the functions of the information processing device. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, a detailed description will be given of an information processing device and an information processing program according to the present application (hereinafter referred to as an "embodiment") with reference to the drawings. Note that the information processing device and the information processing program according to the present application are not limited to the embodiment. Furthermore, the same components in the following embodiments are denoted by the same reference numerals, and duplicated descriptions will be omitted.

[0010] (Embodiment) 1. Introduction Conventionally, there has been known a technique for generating an image description (also called an image caption, hereinafter referred to as an "image description"), which is a sentence that explains the content of an image (still image), from an image. For example, there has been known a technique related to a machine learning model (hereinafter sometimes referred to as an "image caption model") that generates an image description from an image. For example, a visual language model (VLM) is known as such an image caption model. A visual language model 1 is a machine learning model that is trained in advance based on training data that includes pairs of images and image descriptions corresponding to the images. Specifically, the visual language model 1 is a machine learning model that is trained to output an image description corresponding to the image when an image included in the training data is input.

[0011] FIG. 1 is a diagram illustrating a visual language model 1 according to an embodiment. FIG. 1 shows how the visual language model 1 learns based on a pair of an image 20 and an image caption 30 (not shown) corresponding to the image 20. The visual language model 1 processes a sentence in units called tokens. In FIG. 1, the visual language model 1 divides the image caption 30 into k tokens, namely, a first token A1, a second token A2, ..., a kth token Ak (k is a natural number), and reads the first token through the kth token in order. For example, when an image 20 and a start token A0 are input, the visual language model 1 is trained to output the first token A1 that follows the start token A0. For example, the start token A0 may be " <s>". Furthermore, when an image 20, a start token A0, and a first token A1 are input, the visual language model 1 is trained to output a second token A2 that follows the first token A1. Similarly, when a start token A0 and the first token A1 to the (k-1)th token A(k-1) are input, the visual language model 1 is trained to output a k-th token Ak that follows the (k-1)th token A(k-1). In this way, the visual language model 1 is trained to predict the correct next token from an image and given (partial) tokens. When the visual language model 1 finishes generating a sentence, it outputs an end token following the generated sentence. For example, the visual language model 1 may be CoCa (Contrastive Captioners Image-Text Foundation Models), BLIP (Bootstrapping Language-Image Pre-training), BLIP2, GIT (Generative Image to Text Transformer), or the like.

[0012] The information processing device 100 according to the present embodiment improves the visual language model 1 to generate a color visual language model 2, which is a visual language model that generates, from an image, an image description that focuses on the colors of objects included in the image. The color visual language model 2 according to the embodiment is a machine learning model that is an improvement of the visual language model 1, and can be considered a type of visual language model 1. Hereinafter, to distinguish it from the color visual language model 2 according to the embodiment, the visual language model 1 according to the embodiment (i.e., the visual language model 1 before being improved) may be referred to as the visual language model 1 according to the prior art.

[0013] FIG. 2 is a diagram showing an example of an image caption generated by a visual language model 1 according to the prior art. In FIG. 2, the visual language model 1 generates an image caption 31, which is the sentence "A person is washing dishes in a kitchen." shown on the right side of FIG. 2, from an image 21 shown on the left side of FIG. 2. Here, in addition to a dish, image 21 also includes multiple objects (also referred to as objects), such as a kitchen sink and a plant pot placed on a counter. Here, the inclusion of multiple objects in image 21 corresponds to the inclusion of information about each of the multiple objects in image 21. In contrast, visual language model 1 generates an image caption 31 that focuses only on the dish among the multiple objects included in image 21. In other words, the image caption 31 is a concise image caption 31 that includes only information about the dish among the multiple objects included in image 21. Here, the visual language model 1 generating an image description 31 that focuses only on the plate corresponds to the visual language model 1 generating a concise (in other words, poor) image description 31 that focuses only on a specific object without capturing all of the information about multiple objects included in the image 21. That is, the visual language model 1 according to the prior art may generate a concise image description that focuses only on a specific object included in the image, even if the image includes multiple objects. In other words, the visual language model 1 according to the prior art does not necessarily generate a detailed (in other words, rich) image description that touches on each of the multiple objects included in the image, even if the image includes multiple objects. In other words, the visual language model 1 according to the prior art does not necessarily generate a detailed image description that includes information (e.g., a description) about each of the multiple objects included in the image. In contrast, it is desirable for the visual language model 1 to generate a detailed image description that includes information about each of the multiple objects included in the image. That is, it is desirable for the visual language model 1 to generate a more detailed image description corresponding to the image.

[0014] Note that the target in this specification may be a detection target of a visual language model included in an image, such as an object, a person, or a background included in an image.

[0015] FIG. 3 is a diagram illustrating an overview of an image caption generation process by a visual language model 1 according to the prior art. FIG. 3 illustrates an overview of a generation process in which the visual language model 1 generates an image caption corresponding to an image 21 from an image 21. The multiple arrows in FIG. 3 indicate that it is not clear which object among the multiple objects included in the image 21 the attention of the visual language model 1 will be directed to. For example, the visual language model 1 may generate an image caption that focuses only on the faucet among the multiple objects included in the image 21. The visual language model 1 may also generate an image caption that focuses only on the plate among the multiple objects included in the image 21. As such, the visual language model 1 according to the prior art does not necessarily generate a detailed image caption that focuses on each of the multiple objects included in the image, because there is no guarantee that the attention of the visual language model 1 will be directed to each of the multiple objects included in the image.

[0016] 3, the visual language model 1 outputs an end token 301 after the image description 300, which is the sentence "A person is washing dishes," that the visual language model 1 has partially generated based on the image 21. In other words, the visual language model 1 terminates the image description generation process after generating the short image description 300, "A person is washing dishes." This phenomenon is likely to occur when the visual language model 1 is trained to output only one image description from one image. As described above, the visual language model 1 according to the prior art terminates the image description generation process after generating one image description for one image, so the image description tends to be short. Furthermore, a short image description corresponds to a small amount of information contained in the image description. It is desirable to output a description 302, such as "in front of a red plate, ...," or a description 303, such as "in a kitchen," following the image description 300, "A person is washing dishes," that the visual language model 1 has partially generated. In other words, it is desirable for the visual language model 1 to generate longer sentences to generate detailed image descriptions corresponding to the image 21.

[0017] FIG. 4 is a diagram illustrating an overview of an image caption generation process using a color visual language model according to an embodiment. FIG. 4 illustrates an overview of a generation process in which a color visual language model 2 according to an embodiment generates an image caption corresponding to an image 21 from an image 21. The color visual language model 2 according to an embodiment is trained to focus on the colors of objects included in an image. The arrow in FIG. 4 indicates that the attention of the color visual language model 2 is directed toward the red plate included in the image 21. Also, in FIG. 4, the color visual language model 2 generates a caption 302, "in front of a red plate, ...," regarding the red plate included in the image 21, following an image caption 300, "A person is washing dishes," which has been partially generated based on the image 21. The color visual language model 2 according to an embodiment is trained to focus on the colors of objects included in an image and generate an image caption including color information indicating the colors of the objects included in the image. In this way, the color visual language model 2 according to an embodiment is trained to focus on the colors of objects included in an image, thereby generating an image caption focusing on each of the colored objects included in the image. In other words, the color vision language model 2 according to the embodiment can generate a detailed image description that includes information (e.g., a description) about each of the objects having a color included in the image. Furthermore, the color vision language model 2 according to the embodiment is trained to focus on the colors of the objects included in the image, and does not terminate the text generation process until it has generated an image description that mentions each of the objects having a color included in the image. This allows the color vision language model 2 according to the embodiment to generate a detailed image description that focuses on each of the objects having a color included in the image. For example, when an image includes multiple objects, the color vision language model 2 according to the embodiment can generate a detailed image description that describes each of the multiple objects included in the image, as long as each of the multiple objects included in the image has a color.

[0018] FIG. 5 is a diagram illustrating an example of an image caption generated by a color visual language model according to an embodiment. In FIG. 5, the color visual language model 2 generates an image caption 32, which is a sentence shown on the right side of FIG. 5, from the image 21 shown on the left side of FIG. 5: "A person is washing white dishes in a kitchen, to the left is a brown box. On a white pedestal in front of a green plant there are a red plate and a golden bowl." In FIG. 5, the color visual language model 2 focuses on each of the colored objects included in the image, and generates an image caption 32 including a description (also referred to as a depiction or description) of each of the "white dishes," "brown box," "white pedestal," "green plant," "red plate," and "golden bowl." In this way, the color visual language model 2 generates an image caption 32 focusing on each of the multiple colored objects included in the image 21 by focusing on the color of the objects included in the image 21. Here, the color visual language model 2 generating an image caption 32 focusing on each of the multiple objects included in the image 21 corresponds to the color visual language model 2 generating a detailed image caption 32 that describes each of the multiple objects included in the image 21. In other words, when an image includes multiple objects, the color visual language model 2 according to the embodiment can generate a detailed image caption that includes information (e.g., a description) about each of the multiple objects included in the image. Therefore, the color visual language model 2 according to the embodiment can generate a more detailed image caption corresponding to the image.

[0019] 2. Configuration of Information Processing Device An example of the configuration of the information processing device 100 according to the embodiment will be described with reference to Fig. 6. Fig. 6 is a diagram showing an example of the configuration of the information processing device 100 according to the embodiment. The information processing device 100 includes a communication unit 110, a storage unit 120, and a control unit 130.

[0020] (Communication unit 110) The communication unit 110 is realized by a NIC (Network Interface Card), an antenna, etc. The communication unit 110 is connected to various networks by wire or wirelessly, and transmits and receives information to and from other information processing devices other than the information processing device 100, for example.

[0021] (Storage unit 120) The storage unit 120 is realized by, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk. Specifically, the storage unit 120 stores various data. For example, the storage unit 120 stores various programs. For example, the storage unit 120 stores an information processing program according to an embodiment. The storage unit 120 may also store information related to the visual language model 1. The storage unit 120 may also store information related to the attribute visual language model generated by the model generation unit 132. For example, the storage unit 120 may store information related to the color visual language model 2 generated by the model generation unit 132.

[0022] (control unit 130) The control unit 130 is a controller, and is realized, for example, by a CPU (Central Processing Unit) or an MPU (Micro Processing Unit) executing various programs stored in a storage device inside the information processing device 100 using RAM as a work area. The control unit 130 is also a controller, and is realized, for example, by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0023] The control unit 130 has an acquisition unit 131, a model generation unit 132, and a sentence generation unit 133 as functional units, and may realize or execute the information processing actions described below. Note that the internal configuration of the control unit 130 is not limited to the configuration shown in FIG. 6, and may be any other configuration that performs the information processing described below. Also, each functional unit indicates a function of the control unit 130, and does not necessarily have to be physically distinct.

[0024] (Acquisition part 131) The acquisition unit 131 acquires a visual language model 1, which is a machine learning model that generates an image caption, which is a sentence that explains the content of an image, from an image. For example, the acquisition unit 131 may acquire a visual language model, which is a machine learning model that has been trained in advance based on training data including pairs of images and image captions corresponding to the images. For example, the acquisition unit 131 may acquire a visual language model, which is a machine learning model that has been trained to output an image caption corresponding to an image when an image included in the training data is input. For example, the acquisition unit 131 may acquire a visual language model, which is a machine learning model that has been trained to predict the correct next token from an image and given (partially generated) tokens. For example, the acquisition unit 131 may acquire a visual language model, which is a machine learning model that has been trained to estimate the next token from a sequence of tokens currently being generated. For example, the acquisition unit 131 may acquire a visual language model, which is a machine learning model that has been trained to estimate and output the next token from an input image and token sequence. For example, the acquisition unit 131 may acquire a visual language model such as CoCa, BLIP, BLIP2, or GIT. Specifically, the acquisition unit 131 may acquire the visual language model 1 by referring to the storage unit 120. Note that the acquisition unit 131 may acquire the visual language model 1 from an external information processing device via the communication unit 110. When acquiring the visual language model, the acquisition unit 131 may output the visual language model 1 to the model generation unit 132.

[0025] (Model generation unit 132) The model generation unit 132 generates an attribute visual language model, which is a visual language model that generates, from an image, an image caption including attribute information that indicates the attributes of an object included in the image. For example, the attribute information is color information that indicates the color of the object. For example, the model generation unit 132 generates, as an example of the attribute visual language model, a color visual language model, which is a visual language model that generates, from an image, an image caption including color information that indicates the color of an object included in the image.

[0026] FIG. 7 is a diagram illustrating an overview of information processing by the information processing device 100 according to the embodiment. In FIG. 7, the model generation unit 132 may acquire the visual language model 1 output from the acquisition unit 131. When the model generation unit 132 acquires the visual language model 1, the model generation unit 132 may generate a color visual language model 2 by performing reinforcement learning on the visual language model 1 so that image captions generated by the visual language model 1 include color information. Here, reinforcement learning involves trial-and-error learning using three elements (state, action, and reward). Specifically, reinforcement learning involves repeating a process in which, when an agent performs a certain "action" in a certain "state," a "reward" is given if the action is correct. Then, reinforcement learning involves repeating trial-and-error to increase the amount of reward given, thereby acquiring a learning model that can determine appropriate "actions" in various "states."

[0027] In FIG. 7 , the agent in reinforcement learning is the visual language model 1. Furthermore, a "state" in reinforcement learning corresponds to an image input to the visual language model 1. Furthermore, an "action" in reinforcement learning corresponds to whether the visual language model 1 generates an image caption including color information. Furthermore, a correct action in reinforcement learning corresponds to the visual language model 1 generating an image caption including color information. Here, for example, if an image includes a green plant, the visual language model 1 generating an image caption including a description (also referred to as a depiction or description) of a green plant from the image, such as "A green plant...". For example, the model generation unit 132 may repeat a process of providing a positive reward (e.g., 1 point) when the visual language model 1 generates an image caption including color information. Furthermore, the model generation unit 132 may repeat a process of providing no reward (or providing 0 points) when the visual language model 1 generates an image caption without color information. Furthermore, the model generation unit 132 may generate the color visual language model 2 by performing reinforcement learning on the visual language model 1 so that, when an image is input, an image caption corresponding to the image is output, so as to maximize a reward given when an image caption including color information is generated. In this way, the model generation unit 132 may generate the color visual language model 2, which is the visual language model 1 that has been reinforced learned based on a reward given when an image caption including color information is generated. In this way, the model generation unit 132 generates an attribute visual language model (color visual language model in FIG. 7), which is a visual language model that has been reinforced learned based on a reward given when an image caption including attribute information (color information in FIG. 7) is generated.

[0028] (Sentence generation unit 133) The sentence generation unit 133 generates, from the image to be processed, a description of the image to be processed, including attribute information indicating the attributes of an object included in the image to be processed, using the attribute visual language model generated by the model generation unit 132. For example, the sentence generation unit 133 may generate, from the image to be processed, a description of the image to be processed, including color information indicating the color of an object included in the image to be processed, using the color visual language model 2 generated by the model generation unit 132. For example, the sentence generation unit 133 may input the image to be processed into the color visual language model 2 generated by the model generation unit 132, to generate a description of the image to be processed.

[0029] [3. Processing Procedure] FIG. 8 is a flowchart showing the procedure of information processing by the information processing device 100 according to the embodiment. In FIG. 8, the acquisition unit 131 of the information processing device 100 acquires a visual language model (step S11). The model generation unit 132 of the information processing device 100 generates an attribute visual language model, which is a visual language model that generates, from an image, an image caption including attribute information indicating attributes of an object included in the image (step S12). For example, the model generation unit 132 generates, as an example of the attribute visual language model, a color visual language model, which is a visual language model that generates, from an image, an image caption including color information indicating the color of an object included in the image. The sentence generation unit 133 of the information processing device 100 generates a processing target image caption from the processing target image using the attribute visual language model (step S13). For example, the sentence generation unit 133 generates a processing target image caption from the processing target image using a color visual language model, as an example of the attribute visual language model.

[0030] [4. Modifications] The processing according to the above-described embodiment may be implemented in various different forms other than the above embodiment.

[0031] In the above-described embodiment, the model generation unit 132 generates a color visual language model 2, which is a visual language model 1 that has undergone reinforcement learning to include color information. In a modified example, the model generation unit 132 generates a color visual language model 2A by fine-tuning the visual language model 1 based on a dataset that includes a large amount of color information. Here, a dataset that includes a large amount of color information refers to a dataset that includes many pairs of training images and training explanatory text that includes sentences that describe the colors of objects included in the training images. For example, if the training images include green plants, the dataset that includes a large amount of color information may include pairs of training images and training explanatory text that includes a description (also referred to as a depiction or description) of green plants, such as "A green plant..." Furthermore, if the training images include silver faucets, the dataset that includes a large amount of color information may include pairs of training images and training explanatory text that includes a description of silver faucets, such as "A silver faucet..."

[0032] FIG. 9 is a diagram illustrating an overview of information processing by the information processing device 100 according to a modified example. The acquisition unit 131 acquires a dataset including pairs of training images and training explanatory sentences including attribute information indicating attributes of objects included in the training images. For example, the storage unit 120 may store a dataset including pairs of training images and training explanatory sentences including attribute information indicating attributes of objects included in the training images. For example, the storage unit 120 may store a dataset including pairs of training images and training explanatory sentences including color information indicating the colors of objects included in the training images. The acquisition unit 131 may acquire a dataset including pairs of training images and training explanatory sentences including color information indicating the colors of objects included in the training images, by referring to the storage unit 120. When the acquisition unit 131 acquires the dataset, the acquisition unit 131 may output the dataset to the model generation unit 132.

[0033] Furthermore, when a dataset is acquired, the model generation unit 132 may generate a color visual language model 2A, which is a visual language model 1 trained to output training explanatory sentences including color information when a training image included in the dataset is input. In this way, the model generation unit 132 generates an attribute visual language 2A model, which is a visual language model 1 trained to output training explanatory sentences including attribute information (color information in FIG. 9 ), when a training image is input.

[0034] In the above-described embodiment, the attribute information indicating the attributes of an object included in an image is color information indicating the color of the object. However, the attribute information is not limited to color information. For example, the attribute information may be information indicating the shape, material, size, etc. of an object included in an image. The model generation unit 132 may generate an attribute visual language model, which is a visual language model that generates an image caption including, as attribute information, information indicating the shape, material, size, etc. of an object included in an image from an image. If the object included in an image is a person, the attribute information may be information indicating the attributes of the person included in the image. For example, the information indicating the person's attributes may be information indicating the race, age, gender, etc. of the person included in the image. The model generation unit 132 may generate an attribute visual language model, which is a visual language model that generates an image caption including, as attribute information, information indicating the attributes of the person included in the image from an image.

[0035] Furthermore, in the above-described embodiment, an example was described in which the sentence generation unit 133 generates an image caption from an image, but the target for generating the caption is not limited to this. For example, the sentence generation unit 133 may generate a video caption from a video. That is, in one embodiment of the present invention, the image may include a video. A video can be considered a collection of images when viewed frame by frame. Therefore, for example, the sentence generation unit 133 may use an attribute visual language model to generate an image caption for each of multiple frames included in the video, and generate a caption for the entire video based on the image captions corresponding to each of the multiple frames. In this case, the sentence generation unit 133 may acquire a single video caption by summarizing the image captions corresponding to each of the multiple frames using any technology such as a language model.

[0036] [5. Effects] As described above, the information processing device 100 according to the embodiment includes the acquisition unit 131 and the model generation unit 132. The acquisition unit 131 acquires a visual language model, which is a machine learning model that generates, from an image, an image caption that is a sentence that explains the content of the image. The model generation unit 132 generates an attribute visual language model, which is a visual language model that generates, from the image, an image caption that includes attribute information that indicates the attributes of an object included in the image.

[0037] In this way, the information processing device 100 can generate an attribute visual language model that generates an image caption that focuses on each of multiple objects included in an image by focusing on the attributes of the objects included in the image. That is, by using the attribute visual language model, the information processing device 100 can generate a detailed image caption that describes each of the multiple objects included in the image, even if the image includes multiple objects. In other words, by using the attribute visual language model, the information processing device 100 can generate a detailed image caption that includes information (e.g., a caption) about each of the multiple objects included in the image. Furthermore, since the information processing device 100 can generate a detailed image caption that describes each of the objects included in the image from the image, it can contribute to achieving Goal 9 of the Sustainable Development Goals (SDGs), "Build resilient infrastructure, promote inclusive and sustainable industrialization, and promote innovation and infrastructure."

[0038] Furthermore, the model generation unit 132 generates an attribute visual language model, which is a visual language model that has undergone reinforcement learning based on a reward given when an image caption including attribute information is generated.

[0039] As a result, the information processing device 100 can generate detailed image descriptions that explain each of the objects contained in the image from the image, for example, by using an attribute visual language model that has been reinforced learned to generate image descriptions that include attribute information.

[0040] The acquisition unit 131 also acquires a dataset including a set of a training image and a training explanation sentence including attribute information indicating attributes of an object included in the training image. The model generation unit 132 generates an attribute visual language model that is a visual language model trained to output a training explanation sentence when a training image is input.

[0041] This enables the information processing device 100 to generate detailed image descriptions that describe each of the objects contained in the image from the image by using an attribute visual language model that is fine-tuned to generate image descriptions that include attribute information.

[0042] The attribute information is color information that indicates the color of the object.

[0043] As a result, the information processing device 100 can generate a color visual language model that generates an image caption that focuses on each of multiple objects included in an image by focusing on the colors of the objects included in the image. That is, by using the color visual language model, the information processing device 100 can generate, from an image, a detailed image caption that describes each of the multiple objects included in the image, even if the image includes multiple objects. In other words, by using the color visual language model, the information processing device 100 can generate, from an image, a detailed image caption that includes information (e.g., a caption) about each of the multiple objects included in the image.

[0044] The information processing device 100 further includes a sentence generation unit 133. The sentence generation unit 133 uses the attribute visual language model generated by the model generation unit 132 to generate, from the image to be processed, a description of the image to be processed that includes attribute information indicating attributes of an object included in the image to be processed.

[0045] This allows the information processing device 100 to generate a detailed image caption that explains each of a plurality of objects included in the image from the image.

[0046] [6. Hardware Configuration] The information processing device 100 according to the embodiment described above is realized by, for example, a computer 1000 configured as shown in Fig. 10. Fig. 10 is a hardware configuration diagram showing an example of a computer that realizes the functions of the information processing device 100. The computer 1000 includes a CPU 1100, a RAM 1200, a ROM 1300, an HDD 1400, a communication interface (I / F) 1500, an input / output interface (I / F) 1600, and a media interface (I / F) 1700.

[0047] The CPU 1100 operates and controls each unit based on programs stored in the ROM 1300 or the HDD 1400. The ROM 1300 stores a boot program executed by the CPU 1100 when the computer 1000 starts up, programs that depend on the hardware of the computer 1000, and the like.

[0048] The HDD 1400 stores programs executed by the CPU 1100, data used by such programs, etc. The communication interface 1500 receives data from other devices via a predetermined communication network and sends it to the CPU 1100, and transmits data generated by the CPU 1100 to other devices via the predetermined communication network.

[0049] The CPU 1100 controls output devices such as a display and a printer, and input devices such as a keyboard and a mouse, via the input / output interface 1600. The CPU 1100 acquires data from the input devices via the input / output interface 1600. The CPU 1100 also outputs generated data to the output devices via the input / output interface 1600.

[0050] Media interface 1700 reads a program or data stored in recording medium 1800 and provides it to CPU 1100 via RAM 1200. CPU 1100 loads the program or data from recording medium 1800 onto RAM 1200 via media interface 1700 and executes the loaded program. Recording medium 1800 is, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or a PD (Phase Change Rewritable Disc), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory.

[0051] For example, when the computer 1000 functions as the information processing device 100 according to the embodiment, the CPU 1100 of the computer 1000 executes programs loaded onto the RAM 1200 to realize the functions of the control unit 130. The CPU 1100 of the computer 1000 reads and executes these programs from the recording medium 1800, but as another example, the CPU 1100 may obtain these programs from another device via a predetermined communication network.

[0052] Although some of the embodiments of the present application have been described in detail above with reference to the drawings, these are merely examples, and the present invention can be implemented in other forms that include the embodiments described in the Disclosure of the Invention section and that have undergone various modifications and improvements based on the knowledge of those skilled in the art.

[0053] [7. Other] Furthermore, among the processes described in the above embodiments and modifications, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using known methods. In addition, the information including the processing procedures, specific names, various data, and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified. For example, the various information shown in each drawing is not limited to the information shown in the drawings.

[0054] Furthermore, the components of each device shown in the figure are conceptual functional components and do not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc.

[0055] Furthermore, the above-described embodiments and modifications can be combined as appropriate within the scope of not causing any contradiction in the processing content. [Explanation of symbols]

[0056] 100 Information processing device 110 Communications Department 120 Storage section 130 Control Unit 131 Acquisition Department 132 Model Generation Unit 133 Sentence generation section< / s>

Claims

1. an acquisition unit that acquires a visual language model that is a machine learning model that generates an image caption that is a sentence that explains the content of the image from the image; a model generation unit that generates an attribute visual language model, which is the visual language model that generates the image caption including attribute information indicating attributes of an object included in the image from the image; An information processing device comprising:

2. The model generation unit generating the attribute visual language model, which is the visual language model that has been reinforced learned based on a reward given when the image description including the attribute information is generated; The information processing device according to claim 1 .

3. The acquisition unit A data set including a set of a training image and a training description including attribute information indicating an attribute of an object included in the training image is acquired; The model generation unit generating the attribute visual language model, which is the visual language model trained to output the training description when the training image is input; The information processing device according to claim 1 .

4. the attribute information is color information indicating a color of the object; The information processing device according to claim 1 .

5. a sentence generation unit that generates, from the image to be processed, a description of the image to be processed, the description including attribute information indicating attributes of an object included in the image to be processed, using the attribute visual language model generated by the model generation unit; The information processing device according to claim 1 .

6. An acquisition step of acquiring a visual language model, which is a machine learning model that generates an image caption, which is a sentence that explains the content of an image, from an image; a model generation step of generating an attribute visual language model, which is the visual language model for generating the image description from the image, the image description including attribute information indicating attributes of an object included in the image; An information processing program that causes a computer to execute the above.

Images