Learning device and learning method

Abstract

The present application provides a learning device and a learning method that reduce hallucination related to visual information in a model. The learning device (1) includes: an image encoder (11) that encodes an image to generate a first vector; a first text encoder (12) that encodes a prompt to generate a second vector; a learning unit (13) that learns a model (131) that generates a description text of the image using a label assigned to the image, the first vector, and the second vector; a misalignment information acquisition unit (14) that acquires a corrected description text in which a keyword of the description text is corrected and a keyword region corresponding to the corrected keyword; a heat map creation unit (15) that creates a heat map that visualizes an attention region in which attention is paid when the model generates the description text; and an image reward calculation unit (16) that calculates a similarity between the keyword region and the attention region as a first reward, and the learning unit learns the model in such a manner that the first reward increases.

Description

Technical Field

[0001] This disclosure relates to learning devices and learning methods. Background Technology

[0002] Previously, it was known that generative models such as Multimodal Large Language Models (MLLM) were used to generate captions for images. For example, Patent Document 1 discloses a technique in which: the obtained input image is input into a first language generation model that generates captions for an input image of an object being inspected; when an inquiry related to the object being inspected is input, the obtained inquiry is input into a second language generation model that generates a summary of the inquiry; the captions generated by the first language generation model are corrected based on the generated summary; and the corrected captions are input into an image generation model to generate an object image. In addition, Patent Document 1 discloses the following technology: when an image of the object to be inspected is input, the obtained object image is input into an article generation model that generates an article related to the object to be inspected, so as to generate an article related to the object image. The similarity between the article generated by the article generation model and the abstract is calculated, and it is determined whether the calculated similarity is within the allowable range. If it is not within the allowable range, the article generated by the article generation model is corrected, and the corrected article is output to the image generation model, so as to repeatedly perform the processing based on the image generation model.

[0003] Patent Document 1: Japanese Patent Application Publication No. 2021-022007

[0004] However, MLLM cannot learn the relationship between the misaligned parts (the parts where the generated descriptive text does not match the actual image content) and the image based solely on text feedback, so it is difficult to alleviate the visual illusions in MLLM. Summary of the Invention

[0005] The purpose of this disclosure, made in view of this situation, is to provide a learning device and learning method that can alleviate the illusions related to visual information in a model.

[0006] One embodiment of the learning apparatus disclosed herein includes: an image encoder that encodes an image to generate a first vector; a first text encoder that encodes a prompt to generate a second vector; a learning unit that learns a model that generates descriptive text for the image using a label assigned to the image, the first vector, and the second vector; a misalignment information acquisition unit that acquires corrected descriptive text with corrected keywords and a keyword region corresponding to the corrected keywords; a heatmap generation unit that generates a heatmap that visualizes the region of interest that the model focuses on when generating the descriptive text; and an image reward calculation unit that calculates the similarity between the keyword region and the region of interest as a first reward, and the learning unit learns the model in a manner that increases the first reward.

[0007] In one embodiment of the learning method disclosed herein, the learning device performs the following: encoding an image to generate a first vector; encoding a prompt to generate a second vector; learning a model that generates descriptive text for the image using a label assigned to the image, the first vector, and the second vector; obtaining a corrected descriptive text with the keywords of the descriptive text corrected, and a keyword region corresponding to the corrected keywords; creating a heatmap that visualizes the region of interest that the model focuses on when generating the descriptive text; and calculating the similarity between the keyword region and the region of interest as a first reward. The learning includes learning the model in a manner that increases the first reward.

[0008] According to this disclosure, it is possible to alleviate hallucinations related to visual information in the model. Attached Figure Description

[0009] Figure 1 This is a diagram illustrating an example of the configuration of the learning device according to the first embodiment.

[0010] Figure 2 It is a diagram that represents an example of an image displayed by a display device and descriptive text.

[0011] Figure 3 It means Figure 2 The diagram shows a modified example of the descriptive text.

[0012] Figure 4 This is a diagram representing a segmentation example of a modified descriptive text.

[0013] Figure 5 This diagram illustrates the method for calculating rewards when the keyword region has been specified by a marker.

[0014] Figure 6This diagram illustrates the method for calculating rewards when the keyword region has been specified by the bounding box.

[0015] Figure 7 This is a diagram illustrating an example of the configuration of the learning device according to the second embodiment.

[0016] Figure 8 This is a diagram illustrating an example of the configuration of the learning device according to the third embodiment.

[0017] Figure 9 This is a diagram illustrating an example of the configuration of the learning device according to the fourth embodiment.

[0018] Figure 10 This is a flowchart illustrating a sequence example of the learning method of the learning device according to the fourth embodiment.

[0019] Explanation of reference numerals in the attached figures:

[0020] 1, 2, 3, 4… learning device; 10… display device; 11… image encoder; 12… first text encoder; 13… learning unit; 14… misalignment information acquisition unit; 15… heat map production unit; 16… image reward calculation unit; 17… second text encoder; 18… image text reward calculation unit; 19… text reward calculation unit; 131… MLLM. Detailed Implementation

[0021] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

[0022] <First Embodiment>

[0023] Reference Figure 1 The configuration of the learning device according to the first embodiment will be described. Figure 1 The learning device 1 shown includes an image encoder 11, a first text encoder 12, a learning unit 13, a misalignment information acquisition unit 14, a heat map generation unit 15, and an image reward calculation unit 16.

[0024] The image encoder 11 encodes (extracts features) the image input to the learning device 1 to generate a first vector. The image encoder 11 outputs the generated first vector to the learning unit 13 and the heat map generation unit 15.

[0025] The first text encoder 12 encodes the text prompts input to the learning device 1 to generate a second vector. The first text encoder 12 outputs the generated second vector to the learning unit 13. The prompts are, for example, text that instructs the generation of image description text, such as "Please describe what the image depicts."

[0026] The learning unit 13 learns from the Multimodal Large Language Model (MLLM) 131. MLLM 131 is a generative AI that executes instructions based on prompts input to the learning device 1 for an image. In this embodiment, MLLM 131 generates descriptive text for the given image. The learning unit 13 learns from MLLM 131 using the labels assigned to the image, a first vector, and a second vector. The learning unit 13 outputs the descriptive text generated by MLLM 131 to the display device 10.

[0027] Display device 10 displays the images input to learning device 1 and the descriptive text generated by MLLM 131.

[0028] Figure 2 This is a diagram illustrating an example of the image I displayed by the display device 10 and the descriptive text C. Figure 2 In the example shown, there is a misalignment in the descriptive text C of image I. The correct descriptive text is "Two pandas are riding a motorcycle at high speed on a forest road." When descriptive text C contains errors as in this example, the annotator can operate the display device 10 to correct it. In this embodiment, it is possible to correct words (keywords) in descriptive text C and to specify the region within image I corresponding to the corrected keyword (keyword region). Figure 2 In the description text C, editable keywords are shown in bold. Editable keywords can be distinguished from non-keywords by any method, such as changing color, font, or style.

[0029] Figure 3 It means Figure 2 The diagram shows a modified example of the description text C. In Figure 3 In the example shown, the misplaced keywords in the descriptive text C are "one panda," "bicycle," and "city." Therefore, the annotator corrected "one panda" to "two pandas" and designated keyword region 1 and keyword region 2 in image I corresponding to "two pandas." Additionally, the annotator corrected "bicycle" to "motorcycle" and designated keyword region 3 in image I corresponding to "motorcycle." Furthermore, the annotator corrected "city" to "forest" and designated keyword region 4 in image I corresponding to "forest." Annotators can specify keyword regions using tags or bounding boxes. Figure 3 Image I shows the markers for keyword regions 1-4 and the bounding boxes for keyword regions 1 and 3.

[0030] If the description text C has been corrected, the display device 10 stores the corrected description text (corrected description text) C' and the keyword area. The display device 10 outputs the keywords and keyword area to the misalignment information acquisition unit 14.

[0031] The misalignment information acquisition unit 14 acquires the corrected description text C' (after the keywords of the description text have been corrected) and the keyword region corresponding to the corrected keywords from the display device 10. The misalignment information acquisition unit 14 outputs the acquired corrected description text C' to the heatmap generation unit 15. In addition, the misalignment information acquisition unit 14 outputs the acquired keyword region (coordinate values) to the image reward calculation unit 16.

[0032] The heatmap generation unit 15 generates a heatmap that visualizes the region of interest that is the focus of the MLLM131 when generating descriptive text. The heatmap generation unit 15 outputs the region of interest (coordinate values) of the generated heatmap to the image reward calculation unit 16.

[0033] For example, the heatmap generation unit 15 generates a heatmap based on the cross-attention of the first vector input from the image encoder 11 and the corrected descriptive text C' input from the misalignment information acquisition unit 14.

[0034] Figure 4 This is a diagram showing an example of segmentation of the modified description text C'. The heatmap generation unit 15 can also segment the modified description text C' and generate a heatmap for each segmented segment of the modified description text. For example, as... Figure 3 As shown, even if the descriptive text C has been modified, the heatmap generation unit 15 can also be used as follows: Figure 4 As shown, the modified description text C' is divided into modified description text C1 and modified description text C2. In this case, the heatmap generation unit 15 generates a heatmap based on the cross-attention of the first vector and the modified description text C1, and a heatmap based on the cross-attention of the first vector and the modified description text C2.

[0035] The image reward calculation unit 16 calculates the first reward based on the similarity between the keyword region input from the misalignment information acquisition unit 14 and the attention region input from the heatmap production unit 15.

[0036] The learning unit 13 learns the MLLM 131 in a way that increases the first reward calculated by the image reward calculation unit 16. The learning unit 13 updates the parameters of the MLLM, for example, based on an optimization method called DPO (Direct Preference Optimization). The learning unit 13 may also use a separately prepared reward model for reinforcement learning.

[0037] Figure 5This diagram illustrates the method for calculating rewards when the keyword region has been specified by a marker. In the keyword region, such as... Figure 5 As shown, when the region is specified by the marker M, the image reward calculation unit 16 calculates the similarity (first reward) based on the distance between the coordinates of the center point of the marker M and the center point of the region of interest within the heatmap. The shorter the distance between the coordinates, the larger the value of the first reward. The advantage of using markers to specify keyword regions is that, since surrounding information is not mixed in, the object can be accurately specified. However, the definition of the specified keyword region is more ambiguous than when using bounding boxes.

[0038] Figure 6 This diagram illustrates the method for calculating rewards when the keyword region is specified by a bounding box. In the keyword region, such as... Figure 6 As shown, when the keyword region is specified by the bounding box BB, the image reward calculation unit 16 calculates the similarity (first reward) based on the IoU (Intersection over Union) between the bounding box BB and the bounding box of the region of interest within the heatmap. The larger the IoU (i.e., the closer it is to 1), the larger the value of the first reward. Specifying the keyword region by using the bounding box allows for accurate identification of the keyword region, and the comparison with the region of interest is also more accurate. However, unlike specifying using markers, information unrelated to the object may be mixed in within the bounding box.

[0039] <Second Implementation>

[0040] Next, refer to Figure 7 The configuration of the learning device according to the second embodiment will be described. Figure 7 The learning device 2 shown includes an image encoder 11, a first text encoder 12, a learning unit 13, a misalignment information acquisition unit 14, a second text encoder 17, and an image-text reward calculation unit 18. Compared to the learning device 1 of the first embodiment, the learning device 2 of the second embodiment lacks the heat map generation unit 15 and the image reward calculation unit 16, but includes the second text encoder 17 and the image-text reward calculation unit 18. Regarding other configurations, except for the data output destination, they are the same as in the first embodiment; therefore, the same reference numerals are used, and descriptions are appropriately omitted.

[0041] In the learning device 2, the image encoder 11 outputs the first vector to the learning unit 13 and the image-text reward calculation unit 18. Alternatively, the misalignment information acquisition unit 14 may not acquire the keyword region, but instead output the corrected descriptive text to the second text encoder 17.

[0042] The second text encoder 17 encodes the corrected description text input from the misalignment information acquisition unit 14 to generate the third vector. The second text encoder 17 outputs the generated third vector to the image text reward calculation unit 18.

[0043] The image-text reward calculation unit 18 calculates a second reward based on the similarity between the first vector input from the image encoder 11 and the third vector input from the second text encoder 17. For example, the image-text reward calculation unit 18 calculates the similarity (second reward) based on the distance or KL (Kullback-Leibler) divergence between the first and third vectors. The shorter the distance between the vectors, or the smaller the KL divergence, the larger the value of the second reward.

[0044] In the learning device 2, the learning unit 13 learns the MLLM 131 in a way that increases the second reward calculated by the image-text reward calculation unit 18. The learning unit 13 can update the parameters of the MLLM based on an optimization method called DPO, or it can use a separately prepared reward model to perform reinforcement learning.

[0045] <Third Implementation>

[0046] Next, refer to Figure 8 The configuration of the learning device according to the third embodiment will be described. Figure 8 The learning device 3 shown includes an image encoder 11, a first text encoder 12, a learning unit 13, a misalignment information acquisition unit 14, a heat map generation unit 15, an image reward calculation unit 16, a second text encoder 17, and an image-text reward calculation unit 18. Compared to the learning device 1 of the first embodiment, the learning device 3 of the third embodiment differs in that it includes a second text encoder 17 and an image-text reward calculation unit 18. Regarding other configurations, except for the data output destination, they are the same as in the first embodiment; therefore, the same reference numerals are used, and descriptions are appropriately omitted.

[0047] In the learning device 3, the image encoder 11 outputs the first vector to the learning unit 13, the heat map generation unit 15, and the image text reward calculation unit 18. In addition, the misalignment information acquisition unit 14 outputs the acquired corrected description text to the heat map generation unit 15 and the second text encoder 17.

[0048] Similar to the second embodiment, the second text encoder 17 encodes the corrected description text input from the misalignment information acquisition unit 14 to generate the third vector. Additionally, the image text reward calculation unit 18 calculates the similarity between the first vector input from the image encoder 11 and the third vector input from the second text encoder 17 as a second reward.

[0049] In the learning device 3, the learning unit 13 learns the MLLM 131 in a manner that increases the first reward calculated by the image reward calculation unit 16 and the second reward calculated by the image-text reward calculation unit 18. The learning unit 13 may also maximize the average or weighted average of the first and second rewards. The learning unit 13 may update the parameters of the MLLM based on an optimization method called DPO, or it may use a separately prepared reward model to perform reinforcement learning.

[0050] <Fourth Implementation>

[0051] Next, refer to Figure 9 The configuration of the learning device according to the fourth embodiment will be described. Figure 9 The learning device 4 shown includes an image encoder 11, a first text encoder 12, a learning unit 13, a misalignment information acquisition unit 14, a heat map generation unit 15, an image reward calculation unit 16, a second text encoder 17, an image-text reward calculation unit 18, and a text reward calculation unit 19. The learning device 4 according to the fourth embodiment differs from the learning device 3 according to the third embodiment in that it includes a text reward calculation unit 19. Regarding other configurations, except for the data output destination, they are the same as in the third embodiment; therefore, the same reference numerals are used, and descriptions are appropriately omitted.

[0052] In the learning device 3, the learning unit 13 outputs the descriptive text generated by the MLLM 131 to the text reward calculation unit 19. In addition, the misalignment information acquisition unit 14 outputs the acquired corrected descriptive text to the heat map production unit 15, the second text encoder 17, and the text reward calculation unit 19.

[0053] The text reward calculation unit 19 calculates the third reward based on the similarity between the description text input from the learning unit 13 and the corrected description text input from the misalignment information acquisition unit 14.

[0054] In the learning device 3, the learning unit 13 learns the MLLM 131 in a manner that increases the first reward, the second reward, and the third reward. The learning unit 13 may also maximize the average or weighted average of the first reward, the second reward, and the third reward. The learning unit 13 may update the parameters of the MLLM based on an optimization method called DPO, or it may use a separately prepared reward model for reinforcement learning.

[0055] <Learning Methods>

[0056] Next, the learning method based on learning device 4 will be explained. Figure 10 This is a flowchart illustrating a sequence of learning methods.

[0057] In step S101, the image encoder 11 of the learning device 4 encodes the image to generate the first vector.

[0058] In step S102, the first text encoder 12 of the learning device 4 encodes the prompt to generate the second vector.

[0059] In step S103, the learning unit 13 of the learning device 4 uses the first vector and the second vector to learn the MLLM131 of the descriptive text of the generated image.

[0060] In step S104, the annotator determines whether the description text generated by MLLM131 has misalignment. If misalignment occurs, the annotator corrects the description text and designates the keyword region corresponding to the corrected keywords. The learning device 4 then executes step S105 and subsequent processing.

[0061] In step S105, the misalignment information acquisition unit 14 of the learning device 4 acquires the corrected description text and keyword area.

[0062] In step S106, the heat map making unit 15 of the learning device 4 makes a heat map of the region of interest that is focused on when the MLLM 131 generates descriptive text.

[0063] In step S107, the image reward calculation unit 16 of the learning device 4 calculates the similarity between the keyword region and the attention region as the first reward.

[0064] In step S108, the second text encoder 17 of the learning device 4 encodes the modified description text to generate the third vector.

[0065] In step S109, the image-text reward calculation unit 18 of the learning device 4 calculates the similarity between the first vector and the third vector as the second reward.

[0066] In step S110, the text reward calculation unit 19 of the learning device 4 calculates the similarity between the description text and the corrected description text as the third reward.

[0067] In step S111, the learning unit 13 of the learning device 4 learns the MLLM 131 in a manner that increases the first reward, the second reward, and the third reward.

[0068] The processing order of steps S106 and S107, steps S108 and S109, and step S110 is different, and the learning device 4 can also execute these processes in parallel. In the learning method based on learning device 1, learning device 1 only executes the processing of steps S106 and S107 in steps S106 to S110. In the learning method based on learning device 2, learning device 2 only executes the processing of steps S108 and S109 in steps S106 to S110. In the learning method based on learning device 3, learning device 3 only executes the processing of steps S106 to S109 in steps S106 to S110.

[0069] <Variation Example>

[0070] The learning device 1 may also include a text reward calculation unit 19 for calculating the aforementioned third reward. In this case, the learning unit 13 learns the MLLM 131 in a manner that increases both the first and third rewards. Furthermore, in Figure 10 In the modified example of learning device 1, only steps S106, S107, and S110 in steps S106 to S110 are processed.

[0071] The learning device 2 may also include a text reward calculation unit 19 for calculating the aforementioned third reward. In this case, the learning unit 13 learns the MLLM 131 in a manner that increases both the second and third rewards. Furthermore, in Figure 10 In the modified example of learning device 2, only the processes of steps S108, S109, and S110 in steps S106 to S110 are executed.

[0072] As described above, learning devices 1-4, variations of learning device 1, and variations of learning device 2 use images to calculate rewards when misalignment occurs. Therefore, according to the present invention, hallucinations related to visual information can be reduced, thereby improving the performance of MLLM131.

[0073] <Program>

[0074] In order to function as the learning devices 1-4, variations of learning device 1, and variations of learning device 2 described above, a computer capable of executing program commands may also be used. The program causes the computer to perform the aforementioned actions, thereby enabling the computer to function as the learning devices 1-4, variations of learning device 1, and variations of learning device 2.

[0075] Programs can be stored on non-transitory computer-readable media. Examples of non-transitory computer-readable media include flash memory, magnetic recording devices, optical discs, optical-magnetic recording media, or ROM. Program distribution can occur, for example, through the sale, transfer, or lending of portable media such as SD (Secure Digital) cards, DVDs (digital versatile discs), or CD-ROMs (compact disc read-only memory) containing the program. Program distribution can also occur by storing the program in the server's storage and transferring the program from the server to other computers. The program can also be provided as a program product.

[0076] Computers may temporarily store programs stored on portable media or transferred from servers in their storage. The computer's processor then reads the program stored in storage and executes the processing according to the read program. Alternatively, the computer may read the program directly from the portable media and execute the processing according to the program. The computer may also execute the processing according to the received program each time it is transferred from the server. Processing may also be performed using so-called ASP (Application Service Provider) type services, which perform the function solely by executing instructions and obtaining results, without transferring the program from the server to the computer. The program includes information that corresponds to the program, which is used by the computer for processing. For example, data that, while not direct instructions to the computer, has the nature of specifying the computer's processing is considered "information that corresponds to the program."

[0077] The above embodiments have been described as representative examples, but various modifications or alterations can be made without departing from the spirit of this disclosure. For example, multiple constituent modules or processing steps described in the embodiments can be combined into one or one can be divided into multiple.

[0078] Hereinafter, some embodiments of the present disclosure will be illustrated. However, please note that the embodiments of the present disclosure are not limited thereto.

[0079] [Appendix 1]

[0080] A learning device, wherein:

[0081] An image encoder encodes an image to generate a first vector;

[0082] The first text encoder encodes the prompt to generate the second vector;

[0083] The learning unit uses the labels assigned to the above image, the first vector, and the second vector to learn the model that generates the descriptive text of the above image.

[0084] The misalignment information acquisition unit acquires the corrected description text after the keywords of the above description text have been corrected, and the keyword region corresponding to the corrected keywords.

[0085] The heatmap generation section generates heatmaps that visualize the regions of interest during the generation of the descriptive text by the aforementioned model; and

[0086] The image reward calculation unit calculates the similarity between the keyword region and the attention region as the first reward.

[0087] The learning unit learns the model in a way that increases the first reward.

[0088] [Appendix 2]

[0089] In the learning device described in Appendix 1,

[0090] The heatmap generation unit generates the heatmap based on the cross-attention between the first vector and the modified descriptive text.

[0091] [Appendix 3]

[0092] In the learning device described in Appendix 1 or 2,

[0093] The heatmap generation unit segments the modified description text and generates the heatmap for each segmented modified description text.

[0094] [Appendix 4]

[0095] In any of the learning devices described in notes 1 to 3,

[0096] When the keyword region has been specified by a marker, the image reward calculation unit calculates the similarity based on the distance between the coordinates of the center point of the marker and the center point of the region of interest.

[0097] [Appendix 5]

[0098] In any of the learning devices described in notes 1 to 4,

[0099] When the keyword region has been specified by a bounding box, the image reward calculation unit calculates the similarity based on the IoU (Intersection over Union) between the bounding box and the bounding box of the region of interest.

[0100] [Appendix 6]

[0101] A learning device, wherein:

[0102] An image encoder encodes an image to generate a first vector;

[0103] The first text encoder encodes the prompt to generate the second vector;

[0104] The learning unit uses the labels assigned to the above image, the first vector, and the second vector to learn the model that generates the descriptive text of the above image.

[0105] The misalignment information acquisition unit acquires the corrected description text after the keywords of the above description text have been corrected, and the keyword region corresponding to the corrected keywords.

[0106] A second text encoder encodes the aforementioned revised descriptive text to generate a third vector; and

[0107] The image-text reward calculation unit calculates the similarity between the first vector and the third vector mentioned above as the second reward.

[0108] The aforementioned learning unit learns the model in a way that increases the second reward.

[0109] [Appendix 7]

[0110] In the learning device described in Appendix 6,

[0111] The image-text reward calculation unit calculates the similarity based on the distance or KL divergence between the first vector and the third vector.

[0112] [Appendix 8]

[0113] The learning device described in any of the appendices 1 to 5 includes:

[0114] A second text encoder encodes the aforementioned revised descriptive text to generate a third vector; and

[0115] The image-text reward calculation unit calculates the similarity between the first vector and the third vector mentioned above as the second reward.

[0116] The aforementioned learning unit learns the aforementioned model in a manner that increases the aforementioned first reward and the aforementioned second reward.

[0117] [Appendix 9]

[0118] In any of the learning devices described in notes 1 to 5,

[0119] It has a text reward calculation unit that calculates the similarity between the aforementioned descriptive text and the aforementioned revised descriptive text as a third reward.

[0120] The aforementioned learning unit learns the aforementioned model in a manner that increases the aforementioned first reward and the aforementioned third reward.

[0121] [Postscript 10]

[0122] In the learning device described in Appendix 6 or 7,

[0123] It has a text reward calculation unit that calculates the similarity between the aforementioned descriptive text and the aforementioned revised descriptive text as a third reward.

[0124] The aforementioned learning unit learns the aforementioned model in a manner that increases the aforementioned second reward and the aforementioned third reward.

[0125] [Postscript 11]

[0126] In the learning device described in Appendix 8,

[0127] It has a text reward calculation unit that calculates the similarity between the aforementioned descriptive text and the aforementioned revised descriptive text as a third reward.

[0128] The aforementioned learning unit learns the aforementioned model in a manner that increases the aforementioned first reward, the aforementioned second reward, and the aforementioned third reward.

[0129] [Postscript 12]

[0130] A learning method, in which,

[0131] Learning device execution:

[0132] The image is encoded to generate the first vector;

[0133] The prompt is encoded to generate the second vector;

[0134] The model that generates the descriptive text for the above image is learned using the labels assigned to the above image, the first vector, and the second vector.

[0135] Obtain the revised description text after the keywords of the above description text have been corrected, and the keyword area corresponding to the corrected keywords;

[0136] Create a heatmap that visualizes the regions of interest that the model focused on when generating the descriptive text described above; and

[0137] The first reward is calculated based on the similarity between the keyword region and the region of interest.

[0138] The aforementioned learning includes learning the model in a way that increases the first reward.

[0139] [Postscript 13]

[0140] In the learning methods recorded in Appendix 12,

[0141] The heatmap is generated based on the cross-attention between the first vector and the modified descriptive text.

[0142] [Postscript 14]

[0143] In the learning methods described in Appendix 12 or 13,

[0144] The modified description text is segmented, and the heatmap is generated for each segmented modified description text.

[0145] [Postscript 15]

[0146] A learning method, in which,

[0147] Learning device execution:

[0148] The image is encoded to generate the first vector;

[0149] The prompt is encoded to generate the second vector;

[0150] The model that generates the descriptive text of the above image is learned using the labels assigned to the above image, the first vector, and the second vector.

[0151] Obtain the revised description text after the keywords of the above description text have been corrected, and the keyword area corresponding to the corrected keywords;

[0152] The modified description text is encoded to generate the third vector; and

[0153] The similarity between the first vector and the third vector is calculated and used as the second reward.

[0154] The aforementioned learning includes learning the model in a way that increases the second reward.

[0155] [Postscript 16]

[0156] In the learning methods described in Appendix 15,

[0157] The similarity is calculated based on the distance or KL divergence between the first vector and the third vector.

[0158] [Postscript 17]

[0159] In any of the learning methods recorded in notes 12-14,

[0160] The above learning device performs:

[0161] The modified description text is encoded to generate the third vector; and

[0162] The similarity between the first vector and the third vector is calculated and used as the second reward.

[0163] The aforementioned learning includes learning the model in a manner that increases the first reward and the second reward.

[0164] [Postscript 18]

[0165] In any of the learning methods recorded in notes 12-14,

[0166] The learning device described above performs the following: calculates the similarity between the described text and the revised described text, and uses this as the third reward.

[0167] The aforementioned learning includes learning the model in a manner that increases the first reward and the third reward.

[0168] [Postscript 19]

[0169] In the learning methods described in notes 15 or 16,

[0170] The learning device described above performs the following: calculates the similarity between the described text and the revised described text, and uses this as the third reward.

[0171] The aforementioned learning includes learning the model in a way that increases the aforementioned second reward and the aforementioned third reward.

[0172] [Postscript 20]

[0173] In the learning methods described in Appendix 17,

[0174] The learning device described above performs the following: calculates the similarity between the described text and the revised described text, and uses this as the third reward.

[0175] The aforementioned learning includes learning the model in a manner that increases the aforementioned first reward, the aforementioned second reward, and the aforementioned third reward.

Claims

1. A learning device, wherein, have: An image encoder encodes an image to generate a first vector; The first text encoder encodes the prompt to generate the second vector; The learning unit uses the labels assigned to the image, the first vector, and the second vector to learn the model that generates the descriptive text for the image; The misalignment information acquisition unit acquires the corrected description text after the keywords of the description text have been corrected, and the keyword region corresponding to the corrected keywords; The heatmap production unit produces a heatmap that visualizes the regions of interest that the model focuses on when generating the descriptive text. as well as The image reward calculation unit calculates the similarity between the keyword region and the attention region as the first reward. The learning unit learns the model in a manner that increases the first reward.

2. The learning device according to claim 1, wherein, The heatmap generation unit generates the heatmap based on the cross-attention between the first vector and the modified descriptive text.

3. The learning device according to claim 1, wherein, The heatmap generation unit segments the modified description text and generates the heatmap for each segmented modified description text.

4. The learning device according to claim 1, wherein, When the keyword region has been specified by a tag, the image reward calculation unit calculates the similarity based on the distance between the coordinates of the center point of the tag and the center point of the region of interest.

5. The learning device according to claim 1, wherein, When the keyword region has been specified by a bounding box, the image reward calculation unit calculates the similarity based on the IoU (Intersection over Union) between the bounding box and the bounding box of the region of interest.

6. A learning device, wherein, have: An image encoder encodes an image to generate a first vector; The first text encoder encodes the prompt to generate the second vector; The learning unit uses the labels assigned to the image, the first vector, and the second vector to learn the model that generates the descriptive text for the image; The misalignment information acquisition unit acquires the corrected description text after the keywords of the description text have been corrected, and the keyword region corresponding to the corrected keywords; A second text encoder encodes the revised description text to generate a third vector; and The image-text reward calculation unit calculates the similarity between the first vector and the third vector as the second reward. The learning unit learns the model in a manner that increases the second reward.

7. The learning device according to claim 6, wherein, The image-text reward calculation unit calculates the similarity based on the distance or KL divergence between the first vector and the third vector.

8. The learning device according to any one of claims 1 to 5, wherein, have: A second text encoder encodes the revised description text to generate a third vector; and The image-text reward calculation unit calculates the similarity between the first vector and the third vector as the second reward. The learning unit learns the model in a manner that increases the first reward and the second reward.

9. The learning device according to any one of claims 1 to 5, wherein, It includes a text reward calculation unit that calculates the similarity between the descriptive text and the revised descriptive text as a third reward. The learning unit learns the model in a manner that increases the first reward and the third reward.

10. The learning device according to claim 6 or 7, wherein, It includes a text reward calculation unit that calculates the similarity between the descriptive text and the revised descriptive text as a third reward. The learning unit learns the model in a manner that increases the second reward and the third reward.

11. The learning device according to claim 8, wherein, It includes a text reward calculation unit that calculates the similarity between the descriptive text and the revised descriptive text as a third reward. The learning unit learns the model in a manner that increases the first reward, the second reward, and the third reward.

12. A learning method, wherein, Learning device execution: The image is encoded to generate the first vector; The prompt is encoded to generate the second vector; The model that generates the descriptive text for the image is learned using the labels assigned to the image, the first vector, and the second vector. Obtain the revised description text after the keywords of the description text have been corrected, and the keyword region corresponding to the corrected keywords; A heatmap is created that visualizes the region of interest that the model focuses on when generating the descriptive text. as well as The similarity between the keyword region and the region of interest is calculated and used as the first reward. The learning includes learning the model in a manner that increases the first reward.

13. The learning method according to claim 12, wherein, The heatmap is generated based on the cross-attention between the first vector and the modified descriptive text.

14. The learning method according to claim 12, wherein, The modified description text is segmented, and a heatmap is generated for each segmented modified description text.

15. A learning method, wherein, Learning device execution: The image is encoded to generate the first vector; The prompt is encoded to generate the second vector; The model that generates the descriptive text for the image is learned using the labels assigned to the image, the first vector, and the second vector. Obtain the revised description text after the keywords of the description text have been corrected, and the keyword region corresponding to the corrected keywords; Encode the modified description text to generate the third vector; and The similarity between the first vector and the third vector is calculated and used as the second reward. The learning includes learning the model in a manner that increases the second reward.

16. The learning method according to claim 15, wherein, The similarity is calculated based on the distance or KL divergence between the first vector and the third vector.

17. The learning method according to any one of claims 12 to 14, wherein, The learning device performs: Encode the modified description text to generate the third vector; and The similarity between the first vector and the third vector is calculated and used as the second reward. The learning includes learning the model in a manner that increases the first reward and the second reward.

18. The learning method according to any one of claims 12 to 14, wherein, The learning device performs the following: calculating the similarity between the descriptive text and the revised descriptive text as a third reward. The learning includes learning the model in a manner that increases the first reward and the third reward.

19. The learning method according to claim 15 or 16, wherein, The learning device performs the following: calculating the similarity between the descriptive text and the revised descriptive text as a third reward. The learning includes learning the model in a manner that increases the second reward and the third reward.

20. The learning method according to claim 17, wherein, The learning device performs the following: calculating the similarity between the descriptive text and the revised descriptive text as a third reward. The learning includes learning the model in a manner that increases the first reward, the second reward, and the third reward.

Images