Scene graph information generation device, scene graph information generation method, learning device, and learning method

Abstract

The present application relates to a scene graph information generation device, a scene graph information generation method, a learning device, and a learning method, and can efficiently generate an accurate scene graph. A scene graph information generation device (1) includes: an annotated image acquisition unit (11) that acquires a first annotated image in which a plurality of annotations are given to an image; a scene list acquisition unit (14) that acquires a scene list in which a description of an object and a relationship with another object are described for an object within the image by inputting the first annotated image to a first model (10); and a list filtering unit (15) that deletes a description of an erroneous relationship from the scene list.

Description

Technical Field

[0001] This disclosure relates to a scene graph information generation device, a scene graph information generation method, a learning device, and a learning method. Background Technology

[0002] Previously, there was a known technique for generating scene graphs that use graph models to illustrate the relationships between objects in an image. For example, Patent Document 1 discloses a technique that uses a deep learning-based object detection method to detect objects from an image and uses PLSI to detect the context within the image, uses a deep learning-based relation detection method and an ontology method to detect the relationships between objects, and generates a scene graph for an input image.

[0003] Patent Document 1: Japanese Patent Application Publication No. 2021-197129

[0004] However, due to the high difficulty in annotating scene graphs, it is difficult to improve accuracy, and it is also very time-consuming and costly. Therefore, there is a need for a technique to efficiently generate accurate scene graphs. Furthermore, the tuning of multimodal large language models (MLLMs) mainly uses unstructured data; research on tuning MLLMs using structured data has not yet been conducted. Summary of the Invention

[0005] In view of this situation, the purpose of this disclosure is to provide a scene graph information generation apparatus, a scene graph information generation method, a learning apparatus, and a learning method that can efficiently generate accurate scene graphs.

[0006] An embodiment of the present disclosure discloses a scene graph information generation apparatus comprising: an annotated image acquisition unit that acquires a first annotated image to which multiple annotations are assigned; a scene list acquisition unit that inputs the first annotated image into a first model to acquire a scene list describing objects and their relationships with other objects within the image; and a list filtering unit that deletes incorrect descriptions of relationships from the scene list.

[0007] An embodiment of the present disclosure relates to a scene graph information generation method that causes a scene graph information generation apparatus to perform: obtaining a first annotated image to which multiple annotations have been assigned to an image; inputting the first annotated image into a first model to obtain a scene list describing the objects in the image and their relationships with other objects; and deleting incorrect descriptions of relationships from the scene list.

[0008] One embodiment of the learning apparatus disclosed herein includes: a learning unit that learns a model that generates scene graph information of the image using a corrective label assigned to the image; and a second scene graph information acquisition unit that acquires second scene graph information that corrects the scene graph information generated by the model, wherein the corrective label includes the second scene graph information.

[0009] One embodiment of the present disclosure relates to a learning method that causes a learning device to perform: a process of learning a model of scene graph information that generates an image; a process of inputting an image into the model to generate scene graph information; a process of obtaining second scene graph information that has been corrected from the scene graph information generated by the model; and a process of using the second scene graph information to relearn the model.

[0010] According to this disclosure, accurate scene diagrams can be generated efficiently. Attached Figure Description

[0011] Figure 1 This is a block diagram illustrating an example configuration of a scene graph information generation apparatus according to one embodiment.

[0012] Figure 2 This is a diagram representing an example of a list of scenes generated from an annotated image.

[0013] Figure 3 It means Figure 2 The diagram shows a filtering example of the scene list.

[0014] Figure 4 This is a flowchart illustrating an example of the steps of a scene graph information generation method using a scene graph information generation apparatus according to one embodiment.

[0015] Figure 5 This is a block diagram illustrating an example of the configuration of a learning device according to one embodiment.

[0016] Figure 6 This diagram illustrates the processing of the second scene map information acquisition unit in a learning device according to one embodiment.

[0017] Figure 7 This is a flowchart illustrating an example of the steps of a learning method using a learning device according to one embodiment.

[0018] Explanation of reference numerals in the attached figures

[0019] 1…Scene graph information generation device, 2…Learning device, 10…MLLM, 11…Annotated image acquisition unit, 12…Segmentation mask generation unit, 13…Annotation selection unit, 14…Scene list acquisition unit, 15…List filtering unit, 21…Image encoder, 22…Mask encoder, 23…Text encoder, 24…First scene graph information acquisition unit, 25…Learning unit, 26…Second scene graph information acquisition unit, 251…Model. Detailed Implementation

[0020] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

[0021] <Scene Image Information Generation Device>

[0022] Figure 1 This is a block diagram illustrating an example configuration of a scene graph information generation apparatus according to one embodiment. Figure 1 The scene image information generation device 1 shown includes an annotated image acquisition unit 11, a segmentation mask generation unit 12, an annotation selection unit 13, a scene list acquisition unit 14, and a list filtering unit 15.

[0023] The annotated image acquisition unit 11 acquires (collects) a first annotated image, which has been given multiple accurate annotations, as a seed dataset. The annotated image may contain depth information of the image. The annotated image acquisition unit 11 can collect annotated images from datasets such as COCO, LVIS, RefCOCO, and Visual Genome. The annotated image acquisition unit 11 outputs the acquired annotated image to the segmentation mask generation unit 12 and the annotation selection unit 13.

[0024] The segmentation mask generation unit 12 generates a segmentation mask for the first annotated image. The segmentation mask is an image divided into multiple mask regions, and can be generated using, for example, SAM, Semantic-SAM, etc. The segmentation mask generation unit 12 outputs the generated segmentation mask to the annotation selection unit 13.

[0025] The annotation selection unit 13 calculates the similarity between each annotated region (annotation region) within the annotated image input from the annotated image acquisition unit 11 and each mask region of the segmentation mask input from the segmentation mask generation unit 12. If the similarity with any mask region exceeds a threshold, the annotation selection unit 13 retains the annotation corresponding to the annotation region. If the similarity with any mask region is below the threshold, the annotation selection unit 13 deletes the annotation corresponding to the annotation region. The similarity can be an IoU (Intersection over Union) value. The higher the similarity (the closer the IoU value is to 1), the higher the importance of the annotation is considered, and the greater its contribution to scene understanding. Therefore, the annotation selection unit 13 can focus on visually clear and meaningful elements to create a scene graph. The annotation selection unit 13 outputs the processed annotated image (the second annotated image) to the scene list acquisition unit 14.

[0026] The scene list acquisition unit 14 inputs the second-annotated image from the annotation selection unit 13 into an existing, fully trained multimodal large-scale speech model (MLLM) 10, such as GPT4-V. Then, the scene list acquisition unit 14 instructs the MLLM 10 to select a predetermined number of objects from the second-annotated image and generates a scene list describing each selected object and its relationship to other objects. This instruction can be given by the user providing prompts to the MLLM 10. The scene list acquisition unit 14 acquires the scene list generated by the MLLM 10 and outputs it to the list filtering unit 15.

[0027] Figure 2 This is a diagram representing an example of a list of scenes generated from annotated images. Figure 2 The image shows an example where the scene list retrieval unit 14 instructs the MLLM 10 to select five objects from the second annotated image I, and generates a scene list for each object, describing the object's description and its spatial, interactive, functional, social, and emotional relationships with other objects. Additionally, in... Figure 2 To facilitate visual understanding of the positions of each object, the regions of the five objects selected by MLLM10 are enclosed by boxes in the second annotated image I and assigned subject numbers. Figure 2 The right side shows the scene list L generated by MLLM10. The numbers in [ ] refer to the subject numbers.

[0028] The list filtering unit 15 filters (corrects) the scene list input from the scene list acquisition unit 14 according to predetermined rules, deleting erroneous relational descriptions and generating a filtered scene list as scene graph information. The list filtering unit 15 may also use a model different from MLLM10 (e.g., CLIP, LLaVA, or other visual language models) to filter the scene list. In this specification, scene graph information refers to the information needed to create a scene graph. A scene graph is a graph that visualizes the relationships between objects using nodes and edges based on the scene graph information.

[0029] Figure 3 It means Figure 2 The diagram shows a filtering example for the scenario list L. For example, in Figure 2 In the text, for the object with subject number 0, its spatial relationship with other objects is described as "sitting in front of [3], left of

[16] ". This description can be divided into "sitting in front of [3]" and "left of

[16] ". Figure 3 In this way, the description of the relationship is divided according to each object, and the description of the corresponding object is described to the right of the subject number. For example, the description to the right of "[3]" is "blonde-haired woman". Moreover, for the object with subject number 0 (the woman wearing a baseball cap), the description R1 of the relationship that is sitting in front of the object with subject number 3 (the blonde woman) is incorrect, so the description R1 is deleted from the scene list L. For the object with subject number 0, the descriptions R2 to R5 of the relationship are accurate, so they are kept in the scene list L.

[0030] The scene image information generation device 1 may also lack the segmentation mask generation unit 12 and the annotation selection unit 13. In this case, the scene list acquisition unit 14 inputs the first annotated image acquired by the annotated image acquisition unit 11 into the MLLM 10 to obtain the scene list generated by the MLLM 10.

[0031] <Scene Graph Information Generation Method>

[0032] Next, the scene graph information generation method of scene graph information generation device 1 will be explained. Figure 4 This is a flowchart illustrating the steps involved in generating scene graph information.

[0033] In step S101, the annotated image acquisition unit 11 acquires the annotated image.

[0034] In step S102, the segmentation mask generation unit 12 generates a segmentation mask for the annotated image.

[0035] In step S103, the annotation selection unit 13 calculates the IoU between the annotation region in the annotated image and the mask region of the segmentation mask. If the IoU is below a threshold (e.g., 0.5), the annotation corresponding to the annotation region is deleted.

[0036] In step S104, the scene list acquisition unit 14 inputs an annotated image into the MLLM10 and acquires a scene list that describes the objects and their relationships with other objects within the annotated image.

[0037] In step S105, the list filtering unit 15 determines whether there are any erroneous relationships.

[0038] In step S106, the list filtering unit 15 corrects the scene list if there are incorrect relationships, removing the descriptions of incorrect relationships from the scene list. An accurate scene diagram is then generated based on the corrected scene diagram information.

[0039] Thus, according to this disclosure, scene graphs can be generated efficiently by obtaining annotated images. Furthermore, by removing inappropriate relational descriptions from the scene list, high-precision scene graphs can be generated. Additionally, by removing low-importance annotations from the images, even higher-precision scene graphs can be generated.

[0040] <Learning Device>

[0041] Figure 5 This is a block diagram illustrating an example of the configuration of a learning device according to one embodiment. Figure 5 The learning device 2 shown includes an image encoder 21, a mask encoder 22, a text encoder 23, a first scene image information acquisition unit 24, a learning unit 25, and a second scene image information acquisition unit 26.

[0042] Image encoder 21 encodes (extracts features) the image to generate an image vector. Image encoder 21 outputs the generated image vector to learning unit 25.

[0043] The mask encoder 22 encodes the mask image, which masks the image by each region, to generate a mask image vector. The mask encoder 22 outputs the generated mask image vector to the learning unit 25.

[0044] The text encoder 23 encodes the prompts to generate text vectors. The prompts are, for example, text indicating the generation of scene graph information, such as "Please explain the relationship between the objects in the image." The text encoder 23 outputs the generated text vectors to the learning unit 25.

[0045] The first scene map information acquisition unit 24 acquires first scene map information, which is the ground-truth. The first scene map information may be scene map information created by the scene map information generation device 1 described above. The first scene map information acquisition unit 24 outputs the acquired first scene map information to the learning unit 25.

[0046] The learning unit 25 uses the image vector input from the image encoder 21, the mask image vector input from the mask encoder 22, the text vector input from the text encoder 23, and the forward label assigned to the image to learn the model 251 that generates the scene graph information of the image. In the first stage of learning (instruction adjustment), the learning unit 25 uses the first scene graph information of the ground truth as the forward label. The learning unit 25 outputs the scene graph information generated by the model 251 to the second scene graph information acquisition unit 26.

[0047] The second scene graph information acquisition unit 26 uses an existing, fully learned multimodal large-scale language model (MLLM) 10, such as GPT4-V, to correct the scene graph information generated by model 251. The second scene graph information acquisition unit 26 outputs the corrected scene graph information as second scene graph information to the learning unit 25. Among these steps, for... Figure 1 The MLLM and shown Figure 5 The MLLM shown is given the same reference numeral "10", but the two may not be the same model.

[0048] Figure 6 This diagram illustrates the processing of the second scene graph information acquisition unit 26. Model 251 generates scene graph information SG using image I and a mask image M that masks image I by each region. The second scene graph information acquisition unit 26 inputs the scene graph information SG and the mask image M corresponding to the scene graph information SG into MLLM 10. Furthermore, the second scene graph information acquisition unit 26 instructs MLLM 10 to record object labels in more detail, delete incorrect relationships, and add highly relevant relationships. Therefore, MLLM 10 corrects (edits) the scene graph information, generating second scene graph information SG'. Figure 6 In the text, the differences between scene graph information SG and second scene graph information SG' are shown by underline and bold text. For example, MLLM10 removes the incorrect relation "wearing [1] (chair)" for the person with the attached label [2], changes it to the description "wearing a maroon sweater" for [2] (person), and adds the relation "posing for

[13] " and "sitting [1]".

[0049] In the second stage of learning (instruction adjustment), the learning unit 25 uses the second scene graph information SG' as the positive solution label. That is, the learning unit 25 uses image I, a mask image M that masks image I by each region, and the second scene graph information SG' to relearn model 251.

[0050] <Learning Methods>

[0051] Next, the learning method of learning device 2 will be explained. Figure 7 This is a flowchart illustrating the steps involved in a learning method.

[0052] In step S201, image encoder 21 encodes the first image to generate a first image vector. Mask encoder 22 encodes a mask image that masks the first image by each region to generate a first mask image vector. Text encoder 23 encodes a prompt indicating scene graph information for generating the first image to generate a first text vector.

[0053] In step S202, the first scene graph information acquisition unit 24 acquires first scene graph information that describes the relationship between objects in the first image as a correct label for the first image.

[0054] In step S203, the learning unit 25 learns the model 251 that generates scene graph information using the first image vector, the first mask image vector, the first text vector, and the first scene graph information. Alternatively, the learning unit 25 may simultaneously learn the mask encoder 22. Steps S201 to S203 are processed repeatedly using multiple first images.

[0055] In step S204, image encoder 21 encodes the second image to generate a second image vector. Mask encoder 22 encodes a mask image that masks the second image by each region to generate a second mask image vector. Text encoder 23 encodes a prompt indicating scene graph information for generating the second image to generate a second text vector.

[0056] In step S205, model 251 inputs a second image vector, a second mask image vector, and a second text vector to generate scene graph information.

[0057] In step S206, the second scene graph information acquisition unit 26 acquires the second scene graph information by inputting the second mask image, the scene graph information generated by the model 251, and the predetermined rules into a learned model, namely MLLM10, which is different from the model 251.

[0058] In step S207, the learning unit 25 uses the second image vector, the second mask image vector, the second text vector, and the second scene graph information to relearn the model 251. Steps S204 to S207 are processed repeatedly using multiple second images.

[0059] As described above, in the learning device 2, the scene graph information generated by model 251 is corrected, and model 251 is relearned. Therefore, according to this disclosure, structured data can be used to adjust MLLM.

[0060] <Program>

[0061] In order to function as either the scene graph information generation device 1 or the learning device 2 described above, a computer capable of executing program commands can also be used. The program enables the computer to function as either the scene graph information generation device 1 or the learning device 2 by causing the computer to perform the aforementioned actions.

[0062] The program can be pre-stored on a non-transitory computer-readable medium. Examples of non-transitory computer-readable media include flash memory, magnetic recording devices, optical discs, optical-magnetic recording media, or ROM. For example, the program can be distributed by selling, transferring, or renting portable media such as SD (Secure Digital) cards, DVDs (digital versatile discs), or CD-ROMs (compact disc read-only memory) containing the program. The program can also be pre-stored in the server's storage and distributed by transferring the program from the server to other computers. The program can also be provided as a program product.

[0063] The computer may temporarily store programs stored on portable media or transferred from a server in its storage unit. The computer then reads the program stored in storage via its processor and executes the processing based on the read program. The computer may also directly read programs from portable media and execute the processing based on those programs. The computer may execute the processing based on each program transferred from the server sequentially. Alternatively, processing may be performed without transferring programs from the server to the computer, using a so-called ASP (Application Service Provider) type service that functions by executing instructions and obtaining results. A program contains information for the computer to process and that conforms to the program. For example, data that, while not direct instructions to the computer, has the nature of defining the computer's processing is equivalent to "information conforming to the program."

[0064] The above-described embodiments have been presented as representative examples, but various modifications or alterations can be made without departing from the spirit of this disclosure. For example, the constituent blocks or processing steps described in the embodiments can be combined into one or divided into multiple.

[0065] The following examples illustrate some embodiments of the present disclosure. However, it should be noted that the embodiments of the present disclosure are not limited to these embodiments.

[0066] [Appendix 1] A scene graph information generation device, comprising:

[0067] The annotated image acquisition unit acquires a first annotated image to which multiple annotations have been assigned;

[0068] The scene list acquisition unit inputs the first annotated image into the first model and acquires a scene list that describes the objects in the image and their relationships with other objects; and

[0069] The list filtering department removes descriptions of incorrect relationships from the above scenario list.

[0070] [Appendix 2] The scene image information generation device according to Appendix 1 includes:

[0071] The segmentation mask generation unit generates a segmentation mask for the aforementioned image; and

[0072] The annotation selection unit, if the similarity between the annotated region in the image and the masked region of the segmentation mask is below a threshold, outputs a second annotated image that assigns annotations other than the given annotation to the image.

[0073] The above-mentioned scene list acquisition unit replaces the above-mentioned first annotated image with the above-mentioned second annotated image and inputs it into the above-mentioned first model.

[0074] [Appendix 3] The device for generating scene diagram information as described in Appendix 2,

[0075] The above similarity is the IoU value.

[0076] [Appendix 4] A method for generating scene graph information, which causes the scene graph information generation device to perform:

[0077] This process involves obtaining the first annotated image of an image that has been given multiple annotations.

[0078] The process involves inputting the first annotated image into the first model to obtain a list of scenes describing the objects within the image and their relationships with other objects; and...

[0079] This process involves removing descriptions of incorrect relationships from the above list of scenarios.

[0080] [Appendix 5] The scene image information generation device described above shall be executed according to the scene image information generation method described in Appendix 4:

[0081] The process of generating the segmentation mask for the above image; and

[0082] If the similarity between the annotated region in the aforementioned image and the masked region of the aforementioned segmentation mask is below a threshold, the process outputs a second annotated image with additional annotations besides the original annotation.

[0083] The first model is used to replace the first annotated image, and the second annotated image is input instead.

[0084] [Appendix 6] Based on the scene diagram information generation method described in Appendix 5,

[0085] The above similarity is the IoU value.

[0086] [Appendix 7] A learning device comprising:

[0087] The learning unit uses the positive resolution labels assigned to the images to learn the model of the scene graph information that generated the aforementioned images; and

[0088] The second scene graph information acquisition unit acquires second scene graph information that has been corrected from the scene graph information generated by the above model.

[0089] The above correct answer label includes the information from the second scene diagram mentioned above.

[0090] [Appendix 8] The learning device described in Appendix 7 includes:

[0091] An image encoder encodes the above image to generate an image vector;

[0092] A mask encoder encodes a mask image that masks the image by each region to generate a mask image vector; and

[0093] The text encoder encodes the scene graph information indicating the generation of the aforementioned image to generate a text vector.

[0094] The learning unit uses the image vectors, mask image vectors, text vectors, and positive solution labels to learn the model.

[0095] [Note 9] According to the learning device described in Note 7 or 8,

[0096] The second scene map information acquisition unit obtains the second scene map information by inputting a mask image (which masks the image by each region), the scene map information output by the model, and a predetermined rule into a learned model that is different from the model described above.

[0097] [Note 10] Based on any of the learning devices described in Notes 7-9,

[0098] The above correct answer label includes information from the first scene graph.

[0099] The first scene graph information is obtained through a scene graph information generation device, which performs the following:

[0100] This process involves obtaining the first annotated image of an image that has been given multiple annotations.

[0101] The process involves inputting the first annotated image into the first model to obtain a list of scenes describing the objects within the image and their relationships with other objects; and...

[0102] This process involves removing descriptions of incorrect relationships from the above list of scenarios.

[0103] [Note 11] A learning method that causes the learning device to perform:

[0104] This process involves learning the scene graph information of the generated image from a model.

[0105] The process of inputting images into the above model to generate scene graph information;

[0106] The process of obtaining second scene graph information that has been corrected based on the scene graph information generated by the above model; and

[0107] The second scene graph information is used to relearn the model.

[0108] [Appendix 12] Based on the learning methods described in Appendix 11,

[0109] The process of learning the above model includes:

[0110] The process of encoding the first image to generate the first image vector;

[0111] The process of encoding the first mask image, which masks the first image by each region, to generate a first mask image vector;

[0112] The process of encoding the scene graph information indicating the generation of the first image above to generate a first text vector;

[0113] The process of obtaining the first scene graph information as the positive resolution label of the first image; and

[0114] The above-mentioned model for generating scene graph information is learned using the first image vector, the first mask image vector, the first text vector, and the first scene graph information.

[0115] [Appendix 13] Based on the learning methods described in Appendix 11 or 12,

[0116] The process of generating the above scene graph information includes:

[0117] The process of encoding the second image to generate a second image vector;

[0118] The process of encoding the second mask image, which masks the second image by each region, to generate a second mask image vector;

[0119] The process of encoding the scene graph information indicating the generation of the second image to generate a second text vector; and

[0120] The process of inputting the second image vector, the second mask image vector, and the second text vector into the model to generate the scene graph information is as follows.

[0121] [Appendix 14] Based on the learning methods described in Appendix 13,

[0122] The process of obtaining the second scene graph information includes the process of inputting the second mask image, the scene graph information generated by the model, and the predetermined rules into a learned model that is different from the model to obtain the second scene graph information.

[0123] [Appendix 15] Based on any of the learning methods described in Appendices 12-14,

[0124] The process of obtaining the aforementioned first scene image information includes:

[0125] This process involves obtaining a first annotated image that has been given multiple annotations to the aforementioned first image;

[0126] The process of inputting the first annotated image into the first model to obtain a scene list describing the objects in the first image and their relationships with other objects; and

[0127] This process involves removing descriptions of incorrect relationships from the above list of scenarios.

Claims

1. A scene graph information generation device, wherein, have: The annotated image acquisition unit acquires a first annotated image to which multiple annotations have been assigned; The scene list acquisition unit inputs the first annotated image into the first model to obtain a scene list that describes the objects in the image and their relationships with other objects. as well as The list filtering department removes descriptions of incorrect relationships from the above scenario list.

2. The scene graph information generation device according to claim 1, wherein, have: The segmentation mask generation unit generates a segmentation mask for the aforementioned image; and The annotation selection unit, if the similarity between the annotated region in the image and the masked region of the segmentation mask is below a threshold, outputs a second annotated image that assigns annotations other than the given annotation to the image. The above-mentioned scene list acquisition unit replaces the above-mentioned first annotated image with the above-mentioned second annotated image and inputs it into the above-mentioned first model.

3. The scene graph information generation device according to claim 2, wherein, The above similarity is the IoU value.

4. A method for generating scene graph information, wherein, Instruct the scene image information generation device to perform: This process involves obtaining the first annotated image of an image that has been given multiple annotations. The process of inputting the first annotated image into the first model to obtain a list of scenes describing the objects within the image and their relationships with other objects; and This process involves removing descriptions of incorrect relationships from the above list of scenarios.

5. The scene graph information generation method according to claim 4, wherein, The above scene image information generation device shall execute: The process of generating the segmentation mask for the above image; and If the similarity between the annotated region in the aforementioned image and the masked region of the aforementioned segmentation mask is below a threshold, the process outputs a second annotated image that assigns annotations other than the original annotation to the aforementioned image. The first annotated image is replaced by the second annotated image.

6. The scene graph information generation method according to claim 5, wherein, The above similarity is the IoU value.

7. A learning device, wherein, have: The learning department uses the positive resolution labels assigned to the images to learn the model that generated the scene graph information of the above images; and The second scene graph information acquisition unit acquires second scene graph information that has been corrected from the scene graph information generated by the above model. The above correct answer label includes the information from the second scene diagram mentioned above.

8. The learning device according to claim 7, wherein, have: An image encoder encodes the above image to generate an image vector; The mask encoder encodes the mask image, which masks the above image by each region, to generate a mask image vector; as well as The text encoder encodes the scene graph information indicating the generation of the aforementioned image to generate a text vector. The learning unit uses the image vectors, mask image vectors, text vectors, and positive solution labels to learn the model.

9. The learning device according to claim 7, wherein, The second scene map information acquisition unit obtains the second scene map information by inputting a mask image (which masks the image by each region), the scene map information output by the model, and a predetermined rule into a learned model that is different from the model described above.

10. The learning device according to any one of claims 7 to 9, wherein, The above correct answer label includes information from the first scene graph. The first scene graph information is obtained through a scene graph information generation device, which performs the following: This process involves obtaining the first annotated image of an image that has been given multiple annotations. The process of inputting the first annotated image into the first model to obtain a list of scenes describing the objects within the image and their relationships with other objects; and This process involves removing descriptions of incorrect relationships from the above list of scenarios.

11. A learning method, wherein, To enable the learning device to perform: This process involves learning the scene graph information of the generated image from a model. The process of inputting images into the above model to generate scene graph information; The process of obtaining second scene graph information that has been corrected based on the scene graph information generated by the above model; and The second scene graph information is used to relearn the model.

12. The learning method according to claim 11, wherein, The process of learning the above model includes: The process of encoding the first image to generate the first image vector; The process of encoding the first mask image, which masks the first image by each region, to generate a first mask image vector; The process of encoding the scene graph information indicating the generation of the first image above to generate a first text vector; The process of obtaining the first scene graph information as the positive resolution label of the first image; as well as The above-mentioned model for generating scene graph information is learned using the first image vector, the first mask image vector, the first text vector, and the first scene graph information.

13. The learning method according to claim 11, wherein, The process of generating the above scene graph information includes: The process of encoding the second image to generate a second image vector; The process of encoding the second mask image, which masks the second image by each region, to generate a second mask image vector; The process of encoding the scene graph information indicating the generation of the second image to generate a second text vector; as well as The process of inputting the second image vector, the second mask image vector, and the second text vector into the model to generate the scene graph information is as follows.

14. The learning method according to claim 13, wherein, The process of obtaining the second scene graph information includes the process of inputting the second mask image, the scene graph information generated by the model, and the predetermined rules into a learned model that is different from the model to obtain the second scene graph information.

15. The learning method according to any one of claims 11 to 14, wherein, The process of obtaining the aforementioned first scene image information includes: This process involves obtaining a first annotated image that has been given multiple annotations to the aforementioned first image; The process of inputting the first annotated image into the first model to obtain a scene list describing the objects in the first image and their relationships with other objects; and This process involves removing descriptions of incorrect relationships from the above list of scenarios.

Images