Method and control device for culturally-aware image captioning

Abstract

The present specification relates to an image captioning method. This method for culturally-aware image captioning (CIC) according to an embodiment may comprise the steps of: performing image captioning on an image by using an AI model, and acquiring a first image caption for the image; performing cultural visual question answering (VQA) on the image by inputting a cultural question to the AI model, and acquiring a second image caption for the image; inputting the first image caption and the second image caption to a large language model (LLM); and acquiring CIC for the image by inputting a prompt command to the large language model.

Description

Culturally Aware Image Captioning Method and Control Device

[0001] The embodiments relate to a method for culture-aware image captioning and a framework for culture-aware image description.

[0002]

[0003] Recently, the field of image captioning, which generates sentences describing a given image, has seen significant progress in various vision-language tasks, including image captioning, thanks to the emergence of Vision-Language Pretrained Models (VLPs). However, despite these advances, existing VLPs are mostly trained using image-text data available online, resulting in results that do not account for the cultural characteristics of the images.

[0004] In particular, in image captioning tasks, the data used to train existing VLPs not only fails to consider cultural elements of images, but also consists mostly of simple captions for images collected in the West. Therefore, even when images from diverse cultural regions are input, generic captions are generated that do not account for cultural characteristics. As the use of AI systems and applications increases in real-world applications, it is becoming increasingly important to ensure that AI applications do not exhibit discriminatory behavior toward specific groups or populations. Culture distinguishes groups and reflects the lifestyles of individuals within a group. Therefore, as AI systems advance, it is becoming increasingly important to appropriately account for the cultures of diverse groups.

[0005] Research based on prior art called GIVL is well presented in the paper 'GIVL: Improving Geographical Inclusivity of Vision-Language Models with Pre-training Methods'. It pointed out that the performance of existing VLPs varies with Western cultures. To reduce the deviation, they defined cultures as having unique visual features that belong to the same category and similar visual features that belong to different categories. They proposed new pre-training objectives called Image-Knowledge Matching (IKM) and Image Edit Checking (IEC) corresponding to these cultures, and proposed a vision-language model that learns cultural features. This paper shows good performance in geo-diverse vision-language tasks compared to similar-sized VLPs. However, among various vision-language tasks, it has limitations in that it only addresses cultural factors in some tasks such as image classification, image inference, and image-to-text image retrieval, and not in tasks related to image description.

[0006] Data collection has been proposed as another method to reduce cultural bias. For example, in the paper "Visually Grounded Reasoning across Languages ​​and Cultures," representatives from five regions provided images and brief texts. However, existing cultural data lacked image descriptions.

[0007] Therefore, it is necessary to provide a model that expresses the cultural elements expressed in images in order to take cultural diversity into account in image descriptions.

[0008] As a prior art related to the present application, there is Korean Patent Publication No. 10-2019-0140504 (December 20, 2019), “Method and system for generating image captions using reinforcement learning.”

[0009]

[0010] Since there is no fundamental dataset for training AI models for cultural image description, we need to create a dataset for cultural image description. This task requires a lot of human resources, and unlike existing image description datasets, we need to consider various cultures, so we need to recruit people from various cultural regions to carry out the work.

[0011] The purpose of this example is to provide a framework for generating cultural questions, extracting cultural elements from images through Visual Question Answering, and generating captions using BLIP2, one of the VLPs, and generating culturally aware image descriptions using ChatGPT based on the extracted cultural elements.

[0012]

[0013] A method for performing culturally aware image captioning according to one embodiment may include performing image captioning on an image using an AI model and obtaining a first image caption for the image, performing cultural VQA (Visual Question Answering) on ​​the image by inputting a cultural question into the AI ​​model and obtaining a second image caption for the image, inputting the first image caption and the second image caption into a Large Language Model (LLM), and inputting a prompt command into the Large Language Model to obtain a culturally aware image caption (CIC) for the image.

[0014] Additionally, the above prompt command can be set to extract cultural elements from the image.

[0015] Additionally, the above cultural questions can be divided into five categories: architecture, clothing, dance, food, and religion.

[0016] Additionally, if the first image caption does not include at least one of the five categories, it can be assumed that no cultural element exists in the image.

[0017] Additionally, the above cultural questions may be clustered into the above five categories based on questions that are above a pre-specified similarity range.

[0018] According to one embodiment, a deep learning model control device is provided, which controls an AI model and a Large Language Model (LLM) to generate an image caption from an input image, the device including a memory and a processor electrically connected to the memory, wherein the processor is configured to perform image captioning on the image using an AI model to obtain a first image caption for the image, input a cultural question to the AI ​​model to perform cultural VQA (Visual Question Answering) on ​​the image to obtain a second image caption for the image, input the first image caption and the second image caption to the Large Language Model, and input a prompt command to the Large Language Model to obtain a culturally aware image caption (CIC) for the image.

[0019] Additionally, the above prompt command can be set to extract cultural elements from the image.

[0020] Additionally, the above cultural questions can be divided into five categories: architecture, clothing, dance, food, and religion.

[0021] Additionally, if the first image caption does not include at least one of the five categories, it can be assumed that no cultural element exists in the image.

[0022] Additionally, the above cultural questions may be clustered into the above five categories based on questions that are above a pre-specified similarity range.

[0023]

[0024] According to an embodiment, unlike existing image description models, it is possible to provide image descriptions including cultural elements for input images, thereby generating image captions that take cultural diversity into account.

[0025] Examples suggest that AI models can be extended to extract visual elements and describe images not only in cultural contexts but also in other domains, such as art and fashion.

[0026]

[0027] Figure 1 is a flowchart of a cultural awareness image captioning method of an embodiment.

[0028] Figure 2 is a conceptual diagram illustrating the overall flow of the culturally aware image captioning (CIC) framework of the embodiment.

[0029] Figure 3 is a drawing for explaining the usefulness of the cultural awareness image captioning method of the embodiment.

[0030] Figure 4 is a block diagram of a deep learning model control device according to one embodiment.

[0031]

[0032] In describing the embodiments of this specification, if a detailed description of a known technology related to this specification is judged to unnecessarily obscure the gist of this specification, the detailed description will be omitted. In addition, the terms described below are terms defined in consideration of their functions in this specification, and this may vary depending on the intention or custom of the user or operator. Therefore, the definitions should be made based on the contents throughout this specification. The terms used in the detailed description are only for the purpose of describing the embodiments of this specification and should never be construed as limiting. Unless clearly used otherwise, the singular form includes the plural form. In this description, expressions such as "comprises" or "having" are intended to indicate certain features, numbers, steps, operations, elements, parts or combinations thereof, and should not be construed to exclude the presence or possibility of one or more other features, numbers, steps, operations, elements, parts or combinations thereof other than those described.

[0033] Terms containing ordinal numbers, such as "first" and "second," may be used to describe various components, but these components are not limited by these terms. These terms may only be used in a nominal sense to distinguish one component from another, and their ordinal meaning is determined not from the names but from the context of the description.

[0034] The term “and / or” is used to include any combination of the multiple items being referred to. For example, “and / or B” means all three cases, “”“”“and B”.

[0035] When it is said that a component is "connected" or "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but there may also be other components in between.

[0036] Hereinafter, specific embodiments of the present invention will be described with reference to the drawings. The following detailed description is provided to facilitate a comprehensive understanding of the methods, devices, and / or objects described herein. However, this is merely an example, and the present invention is not limited thereto.

[0037] Figure 1 is a flowchart of a cultural awareness image captioning method of an embodiment.

[0038] Referring to FIG. 1, the culturally aware image captioning method of the embodiment may include a step of obtaining a first image caption for an image (S110), a step of obtaining a second image caption for the image (S120), a step of inputting the first image caption and the second image caption into a large language model (LLM) (S130), and a step of inputting a prompt command into the large language model to obtain a culturally aware image caption (CIC) for the image (S140).

[0039] The step (S110) of obtaining a first image caption is a step of obtaining a general image caption for a single image. That is, the first image caption of the embodiment refers to an image caption generated using a previously used AI model for a single image.

[0040] The step of obtaining a second image caption for an image (S120) is a step of inputting a cultural question into an AI model to perform VQA (Visual Question Answering) on ​​one image and obtaining a second image caption for one image.

[0041] Visual Question Answering (VQA) is an artificial intelligence technology that enables AI models to understand questions about images and generate answers based on those images. Combining computer vision and natural language processing, VQA can interpret visual data into text and provide appropriate answers to natural language questions.

[0042] Cultural questions are questions generated by an AI model or processor based on the cultural category of an image, designed to more effectively extract cultural and visual elements from a single image. For example, the cultural questions in this example can be divided into five categories: architecture, clothing, dance or music, food or drink, and religion.

[0043] The second image caption of the embodiment means performing VQA using cultural questions on one image and obtaining the answer.

[0044] The step of inputting the first image caption and the second image caption into a Large Language Model (LLM) (S130) and the step of inputting a prompt command into the Large Language Model (LLM) to obtain a culturally aware image caption (CIC) for an image (S140) are steps of inputting the first image caption and the second image caption generated above into a Large Language Model such as chat-GPT and inputting a prompt command into the Large Language Model to generate a culturally aware image caption for one image.

[0045] A culturally aware image caption is an image caption obtained by more effectively extracting cultural elements from an image than an image caption obtained by performing conventional image captioning on a single image as a result of performing a culturally aware image captioning method of an embodiment on a single image.

[0046] The first image caption and the second image caption can be input to the large language model regardless of the temporal order.

[0047] Referring to FIG. 2 below, the cultural awareness image captioning method of the embodiment and the overall framework of the embodiment are described in more detail.

[0048] Figure 2 is a conceptual diagram illustrating the overall flow of the culturally aware image captioning (CIC) framework of the embodiment.

[0049] The Caption Model in Fig. 2 may be referred to as the image caption model of the embodiment. The Culture VQA in Fig. 2 may be referred to as the VQA model of the embodiment. The LLM in Fig. 2 may be referred to as the large language model of the embodiment.

[0050] In FIG. 2, the Captioning prompt may be referred to as the first image caption of the embodiment, and the VQA results may be referred to as the second image caption of the embodiment.

[0051] In this example, (1) an AI model or processor is used to generate cultural questions based on cultural categories (e.g., five categories: architecture, clothing, dance, music, and food); (2) the generated cultural questions are used to perform visual question answering (VQA) on the AI ​​model to extract cultural visual elements for an image; and (3) LLM is used with prompts to generate culturally aware captions.

[0052] Cultural questions are pre-generated questions designed to extract cultural elements from images. For example, cultural questions can be categorized into five categories: architecture, clothing, dance or music, food or drink, and religion.

[0053] In one embodiment, questions unrelated to cultural categories can be preemptively removed by an AI model or processor. Table 1 presents an example of removing questions unrelated to cultural categories and clustering questions.

[0054] categoryQuestionClusterArchitecture1237Clothing2039Food & Drink2254Dance & Music3516Religion1525total1,05431

[0055] In one embodiment, cultural questions can be clustered based on a predefined similarity level (e.g., 90%), with each cluster containing at least eight questions. For example, multiple groups can be created for each category, and each group may not include questions related to multiple cultural elements. This means, for example, that questions about architecture will not include questions related to other categories.

[0056] Afterwards, the generated cultural questions can be used to extract cultural visual elements from images, for example, using VQA from BLIP2 (Bootstrapped Language-Image Pre-training 2). BLIP2 is a vision-language pre-training (VLP) model developed to enhance the ability of AI models to integrate and understand visual and textual data.

[0057] However, cultural visual elements for the five cultural categories exemplified above rarely appear simultaneously in real images. Therefore, if the VQA results for all previously generated cultural questions are fed to a large language model, the large language model may generate captions describing cultural elements that do not exist in the image, a hallucination problem. To address this, the embodiment assumes that a cultural element exists in a given input image if the first image caption contains at least one word related to the cultural category. In other words, the embodiment assumes that a cultural element does not exist in a given input image if the first image caption does not contain at least one word related to the cultural category.

[0058] In Figure 2, the first image caption, "Two Asian men sitting on a bench eating," extracted the words "bench," "Asia," "male," and "eating" from the architecture, clothing, and food and beverage related words. Questions related to these cultural elements can be used to generate VQA results, i.e., the second image caption.

[0059] Below, the prompt commands of the embodiment are described.

[0060] The prompt command of the embodiment is set to input the first image caption and the second image caption to the LLM so that the LLM extracts cultural elements.

[0061] For example, the prompt command could be set to "I'll tell you the VQA results. Please change the caption based on the VQA results. When changing the caption, do not simply append the VQA results to the caption; use all VQA results. In other words, do not skip any information."

[0062] The prompt command in the example included the phrase, "Do not simply append the VQA results to the caption," to enable LLM to generate culturally-aware image captions in a single sentence. Furthermore, the prompt command in the example included the phrase, "Use all VQA results," to prevent LLM from arbitrarily ignoring VQA results.

[0063] Figure 3 is a drawing for explaining the usefulness of the cultural awareness image captioning method of the embodiment.

[0064] Figure 3 illustrates the differences in captions generated from four models including a culture-aware image captioning method or a framework for culture-aware image description of an embodiment.

[0065] Culturally Aware Image Captioning (CIC), an example, generates captions to describe cultural visual elements not captured by captions generated by existing VLPs (e.g., GIT, CoCa, BLIP2). For example, while existing models can describe spaces lacking cultural visual elements like "hallway" or "bed," CIC can describe the architectural style of a given cultural group. CIC can also describe the country to which the cultural group belongs.

[0066] Referring to FIG. 3, for example, it can be seen that the CIC of the embodiment describes words such as dress, baroque-style setting, gowns, and coat, which are cultural visual elements in Western images.

[0067] For example, the CIC of the embodiment can be seen describing words such as architectural style of Karachi and kurta pyjama, which are cultural visual elements in the image of South Asia.

[0068] For example, the CIC of the embodiment can be seen describing words such as style of the buildings being African, which are cultural visual elements in the image of Africa.

[0069] Additionally, we conducted a first survey to determine how well the cultural elements described by CIC matched the perceptions of people in that cultural region.

[0070] Table 2 presents the agreement between participants' responses and the cultural categories extracted from the framework of the examples.

[0071] ArchitectureClothingFood & DrinkDance & MusicReligion0.640.960.780.840.64

[0072] Referring to Table 2, the Clothing category showed the highest match rate, with an overall match rate of over 60%.

[0073] Existing image caption models fail to describe the background surrounding an image in detail, and user survey results show that their accuracy for cultural visual elements such as architecture and religion is relatively low.

[0074] A second survey evaluated how well captions generated by the framework described in the example described cultural aspects compared to captions generated by existing image captioning models. Participants rated the captions based on the quality of their cultural descriptions.

[0075] ModelWestSouth AsiaAfricaEast AsiaGIT0.080.10.170.16CoCa0.10.210.170.15BLIP20.20.140.210.26CIC(ours)0.620.550.440.43

[0076] Table 3 shows that the framework in this example scored significantly higher than other existing models across all four cultural groups—West, South Asia, Africa, and East Asia—by approximately 17% to 42%. Based on the user survey results, we can conclude that the framework in this example excels at generating cultural descriptions from images that include cultural elements.

[0077] Figure 4 is a block diagram of a deep learning model control device according to one embodiment.

[0078] Referring to FIG. 4, the deep learning model control device of the embodiment may include a processor (420) and a memory (430).

[0079] Figure 4 shows a hardware-wise reconstruction of a deep learning model control device for generating culturally aware image captions according to an embodiment.

[0080] The processor (420) may be configured to perform image captioning using an AI model for one image to obtain a first image caption for the image, input a cultural question into the AI ​​model to perform cultural VQA (Visual Question Answering) for the image to obtain a second image caption for the image, input the first image caption and the second image caption into the large language model, and input a prompt command into the large language model to obtain a culturally aware image caption (CIC) for the image.

[0081] According to one embodiment, the memory (430) can store at least one deep learning model among an image caption model, a VQA model, or a large language model in a volatile memory (random access memory, RAM) by the processor (420).

[0082] According to one embodiment, the memory (430) can store the processing result of the volatile memory in a non-volatile memory (read only memory, ROM).

[0083] Embodiments according to the present specification may be implemented by various means, for example, hardware, firmware, software, or a combination thereof. In the case of hardware implementation, an embodiment of the present specification may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processor controllers, microcontrollers, microprocessors, etc. In the case of firmware or software implementation, an embodiment of the present specification may be implemented in the form of a module, procedure, function, etc. that performs the capabilities or operations described above. Software code may be stored in the memory (430) and executed by the processor (420). The memory (430) may be located inside or outside the processor (420) and may exchange data with the processor (420) by various means already known.

[0084] Meanwhile, the embodiments can be implemented as computer-readable code on a computer-readable recording medium. A computer-readable recording medium includes any type of recording device that stores data that can be read by a computer system.

[0085] Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disks, and optical data storage devices. Furthermore, the computer-readable recording media can be distributed across network-connected computer systems, allowing computer-readable code to be stored and executed in a distributed manner. Furthermore, functional programs, codes, and code segments for implementing the embodiments can be readily inferred by programmers in the technical field to which the embodiments pertain.

[0086] The description of this specification above is provided for illustrative purposes only, and those skilled in the art will readily appreciate that the embodiments described herein can be readily modified into other specific forms without altering the technical spirit or essential characteristics of the embodiments. Therefore, the embodiments described above should be understood to be illustrative in all respects and not restrictive.

[0087] The scope of the embodiments is indicated by the claims described below rather than the detailed description above, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the embodiments.

Claims

1. A step of performing image captioning using an AI model for an image and obtaining a first image caption for the image; A step of inputting a cultural question into the AI ​​model to perform cultural VQA (Visual Question Answering) on ​​the image and obtaining a second image caption for the image; A step of inputting the first image caption and the second image caption into a large language model (LLM); and A culturally aware image captioning method comprising a step of obtaining a culturally aware image caption (CIC) for the image by inputting a prompt command to the large language model.

2. In paragraph 1, A culture-aware image captioning method wherein the above prompt command is set to extract cultural elements from the image.

3. In paragraph 1, The above cultural questions are divided into five categories: architecture, clothing, dance, food, and religion. A method for captioning culturally aware images.

4. In paragraph 3, A culturally aware image captioning method that assumes that no cultural element exists in the image if the first image caption does not include at least one of the five categories.

5. In paragraph 3, The above cultural questions are clustered into the above five categories based on questions with a similarity level above a pre-specified range, and are a culturally aware image captioning method.

6. In a device that controls an AI model and a Large Language Model (LLM) to generate an image caption from an input image, memory; and A processor electrically connected to the above memory, The above processor, Perform image captioning on the above image using an AI model to obtain a first image caption for the image, By inputting a cultural question into the AI ​​model, a cultural VQA (Visual Question Answering) is performed on the image to obtain a second image caption for the image, Inputting the first image caption and the second image caption into the large language model, A control device for culturally aware image captioning, configured to obtain culturally aware image captioning (CIC) for the image by inputting a prompt command to the above-mentioned large language model.

7. In paragraph 6, The above prompt command is a control device for culture-aware image captioning, which is a command set to extract cultural elements from the image.

8. In paragraph 6, The above cultural questions are a control device for cultural awareness image captioning divided into five categories: architecture, clothing, dance, food, and religion.

9. In paragraph 8, A control device for culturally aware image captioning that assumes that no cultural element exists in the image if the first image caption does not include at least one of the five categories.

10. In paragraph 8, The above cultural questions are a control device for culture-aware image captioning, where the questions are clustered into the above five categories based on questions that are above a pre-specified similarity range.

Images