Method and system for generating image caption using image classification and object detection

Abstract

Disclosed are a method and a system for generating an image caption using image classification and object detection. A method for generating an image caption according to one embodiment may comprise the steps of: generating a result of image classification or object detection for an input image; generating a prompt for a language model on the basis of the generated result; and generating an image caption for the image by inputting the generated prompt into the language model.

Description

Method and system for generating image captions using image classification and object recognition

[0001] The description below relates to a method and system for generating image captions using image classification and / or object detection.

[0002] For image captioning, a Vision Language Model (VLM) is typically used to generate captions for images. A VLM typically includes an image encoder that receives an image as input and generates an embedding, a language model (e.g., a Large Language Model (LLM)) that receives text as input and generates text as an image caption, and a module (e.g., a lightweight querying transformer (Q-Former)) that harmonizes the image embedding generated by the image encoder with a text embedding separately input by the user.

[0003] This recent image captioning technique is differentiated from existing image labeling techniques that generate word-level labels, captions, and descriptions in that it can provide one or more sentence-like descriptions of the input image as captions using a language model.

[0004] Thus, the VLM for image captioning is fundamentally much larger than the LLM because it combines an image encoder and additional modules in addition to the LLM. The image encoder is responsible for receiving and understanding images as input, making it optimized for images. However, this requires significant processing time and capacity.

[0005] Existing VLMs have limitations, such as their large capacity and the limitations of on-device memory and speed, making them extremely difficult to utilize for on-device machine learning (ML). For example, existing VLMs typically have model sizes exceeding 4-5 GB, and processing a single image on ML equipment like the NVIDIA V100 is extremely slow, taking less than 1,000 ms.

[0006] A method and system for generating image captions using image classification and / or object detection are provided.

[0007] A method for generating an image caption for a computer device including at least one processor, the method comprising: generating a result of image classification or object recognition for an input image by the at least one processor; generating a prompt for a language model based on the generated result by the at least one processor; and generating an image caption for the image by inputting the generated prompt into the language model by the at least one processor.

[0008] According to one aspect, the step of generating the result may be characterized by generating the output of one or more image classification models for the image classification as the result, generating the output of one or more object recognition models for the object recognition as the result, or generating the output of one or more image classification models and the output of one or more object recognition models as the result.

[0009] According to another aspect, the step of generating the result may be characterized by generating a first output of one or more image classification models and a second output of one or more object recognition models as the result, and the step of generating the prompt may be characterized by merging the first output and the second output into one result and generating the prompt using the merged result.

[0010] According to another aspect, the step of generating the prompt may be characterized by inputting the generated result into the language model and generating the prompt using the output of the language model.

[0011] According to another aspect, the image caption generation method may further include a step of comparing a first image caption generated by inputting the image into a Vision Language Model (VLM) with a second image caption generated by inputting the prompt into the language model to modify the prompt; and a step of inputting the modified prompt into the language model to generate an additional image caption for the image.

[0012] According to another aspect, the step of modifying the prompt may be characterized by including the steps of: transmitting the image to a server; receiving the first image caption generated by inputting the image into a VLM (Vision Language Model) implemented on the server from the server; and modifying the prompt by comparing the first image caption and the second image caption.

[0013] According to another aspect, the language model may be characterized by including a language model implemented in an on-device form in the computer device or an operating system of the computer device.

[0014] According to another aspect, the step of generating the image caption may include a step of generating a LoRA model by fine-tuning the language model according to a predetermined purpose through LoRA (Low-Rank Adaptation) learning for the language model; and a step of generating an image caption for the image by inputting the generated prompt into a Large Language Model (LLM) to which the generated LoRA model is applied.

[0015] According to another aspect, the step of generating the evaluation prompt may be characterized by generating the evaluation prompt based on evaluation criteria of a specific task to be performed by a large language model (LLM).

[0016] According to another aspect, the predetermined use may be characterized by including a predetermined language.

[0017] According to another aspect, the step of generating the LoRA model may be characterized by using the first image generation result generated by inputting the image into the VLM as learning data for the LoRA learning to fine-tune the language model.

[0018] According to another aspect, the VLM may be implemented on a server, and the learning data may be generated and transmitted using the VLM on the server.

[0019] A computer program stored on a computer-readable recording medium is provided for executing the above method on a computer device in combination with a computer device.

[0020] A computer-readable recording medium having recorded thereon a program for executing the above method on a computer device is provided.

[0021] A computer device is provided, comprising at least one processor implemented to execute computer-readable instructions, characterized in that the at least one processor generates a result of image classification or object recognition for an input image, generates a prompt for a language model based on the generated result, and inputs the generated prompt to the language model to generate an image caption for the image.

[0022] A method and system for generating image captions using image classification and / or object detection can be provided.

[0023] FIG. 1 is a block diagram illustrating an example of a computer device according to one embodiment of the present invention.

[0024] FIG. 2 is a diagram illustrating an example of an image caption generation system according to one embodiment of the present invention.

[0025] FIG. 3 is a diagram illustrating an example of a model for object recognition and / or image classification according to one embodiment of the present invention.

[0026] FIGS. 4 and 5 are drawings illustrating examples of a process for generating an image caption according to one embodiment of the present invention.

[0027] FIGS. 6 and 7 are drawings illustrating another example of the process of generating an image caption according to one embodiment of the present invention.

[0028] FIG. 8 is a diagram illustrating an example of a LoRA model according to one embodiment of the present invention.

[0029] FIG. 9 is a flowchart illustrating an example of a method for generating an image caption according to one embodiment of the present invention.

[0030] Hereinafter, the detailed description will be given with reference to the attached drawings.

[0031] An image caption generation system according to embodiments of the present invention may be implemented by at least one computer device. In this case, a computer program according to an embodiment of the present invention may be installed and run on the computer device, and the computer device may perform an image caption generation method according to embodiments of the present invention under the control of the run computer program. The above-described computer program may be stored on a computer-readable recording medium so as to be coupled to the computer device and cause the computer to execute the image caption generation method.

[0032] FIG. 1 is a block diagram illustrating an example of a computer device according to an embodiment of the present invention. As illustrated in FIG. 1, the computer device (100) may include a memory (110), a processor (120), a communication interface (130), and an input / output interface (I / O interface) 140. The memory (110) is a computer-readable recording medium and may include a random access memory (RAM), a read only memory (ROM), and a permanent mass storage device such as a disk drive. Here, the ROM and the permanent mass storage device such as the disk drive may be included in the computer device (100) as a separate permanent storage device distinct from the memory (110). In addition, an operating system and at least one program code may be stored in the memory (110). These software components may be loaded into the memory (110) from a computer-readable recording medium separate from the memory (110). This separate computer-readable recording medium may include a computer-readable recording medium such as a floppy drive, a disk, a tape, a DVD / CD-ROM drive, a memory card, etc. In another embodiment, the software components may be loaded into the memory (110) via a communication interface (130) other than a computer-readable recording medium. For example, the software components may be loaded into the memory (110) of the computer device (100) based on a computer program installed by files received via a network (Network, 160).

[0033] The processor (120) may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input / output operations. Instructions may be provided to the processor (120) via the memory (110) or the communication interface (130). For example, the processor (120) may be configured to execute instructions received according to program code stored in a storage device such as the memory (110).

[0034] The communication interface (130) may provide a function for the computer device (100) to communicate with other devices via a network (160). For example, requests, commands, data, files, etc. generated by the processor (120) of the computer device (100) according to program codes stored in a recording device such as a memory (110) may be transmitted to other devices via the network (160) under the control of the communication interface (130). Conversely, signals, commands, data, files, etc. from other devices may be received by the computer device (100) via the communication interface (130) of the computer device (100) via the network (160). Signals, commands, data, etc. received via the communication interface (130) may be transmitted to the processor (120) or the memory (110), and files, etc. may be stored in a storage medium (the aforementioned permanent storage device) that the computer device (100) may further include.

[0035] The input / output interface (140) may be a means for interfacing with an input / output device (I / O device, 150). For example, the input device may include a device such as a microphone, a keyboard, or a mouse, and the output device may include a device such as a display or a speaker. As another example, the input / output interface (140) may be a means for interfacing with a device that integrates input and output functions, such as a touchscreen. The input / output device (150) may also be configured as a single device with the computer device (100).

[0036] Additionally, in other embodiments, the computer device (100) may include fewer or more components than those illustrated in FIG. 1. However, it is not necessary to explicitly illustrate most conventional components. For example, the computer device (100) may be implemented to include at least some of the input / output devices (150) described above, or may further include other components such as a transceiver, a database, etc.

[0037] As previously explained, the Vision Language Model (VLM) for image captioning is fundamentally much larger than a language model, as it consists of a language model like the Large Language Model (LLM), an image encoder, and additional modules. In particular, while language models are increasingly being built into on-device devices or mobile operating systems, the use of image encoders poses a significant hurdle for processing image captioning on-device, such as mobile devices.

[0038] On the other hand, machine learning models based on image classification (hereinafter referred to as "image classification models") or object detection (hereinafter referred to as "object recognition models") have been developed since the early days of deep learning, so their performance is well optimized and they are considerably simpler than recent transformer models. Image classification models can be implemented to output information about labels and probabilities by labeling the content of the input image, and object recognition models can be implemented to output information about labels and label locations by labeling the location of objects within the image. Furthermore, many of these image classification models and object recognition models have been lightweighted for mobile environments, and many models employ lightweight techniques such as weight quantization and pruning. For example, image classification models and object recognition models are less than 50 MB in size and have processing speeds of less than 100 ms on smartphones. Representative models include "Efficientdet," "Mobilenet," and "Yolo."

[0039] Embodiments of the present invention can provide a method and system for generating image captions using image classification and / or object recognition.

[0040] FIG. 2 is a diagram illustrating an example of an image caption generation system according to one embodiment of the present invention. The image caption generation system (200) according to this embodiment may be implemented by a computer device (100) and may include a model (210) for object recognition and / or image classification, a prompt processing unit (220), and a language model (230).

[0041] The model (210) for object recognition and / or image classification may include at least one image classification model and / or at least one object recognition model. In other words, the image caption generation system (200) may utilize the model (210) for object recognition and / or image classification, which includes at least one image classification model and / or at least one object recognition model, instead of the image encoder of the VLM, which previously required a large capacity and high speed. In this case, the image caption generation system (200) may generate an image caption including a description of an input image (240) by comprehensively utilizing the model (210) for object recognition and / or image classification and the language model (220). Through this, the image caption generation system (200) may utilize the fast and lightweight processing properties of the image classification model and / or the object recognition model.

[0042] An object recognition model can recognize an object in an image (240) and output the class of the object and its location within the image (240) as a result. For example, the object recognition model can input an image (240) containing a human figure and output the class "human" and the coordinates of the object corresponding to the class "human" within the image (240). The object recognition model can also output the class and location for each of two or more objects in a single image (240).

[0043] An image classification model can output at least one label indicating the content of the image (240) and a probability for each label. These labels can also be expressed as classes.

[0044] The prompt processing unit (220) can generate a prompt for the language model (230) by changing the results of each model included in the object recognition and / or image classification model (210) into a certain text template format. At this time, the prompt processing unit (220) can improve the prompt so that the results of the image classification model and the results of the object recognition model are well combined. The prompt generated by the prompt processing unit (220) can be transmitted to the language model (230).

[0045] The language model (230) can generate an image caption (250) that includes at least one sentence describing the content of the image (240) according to the transmitted prompt.

[0046] According to an embodiment, the image caption generation system (200) may not directly include a language model (230), but may utilize a language model implemented in the computer device (100) or included in the operating system of the computer device (100).

[0047] Meanwhile, if the image caption generation system (200) includes a language model (230), it may include a model to which LoRA (Low-Rank Adaptation) learning has been applied to increase the accuracy of the results. In this case, LoRA learning for the language model (230) may be performed using unsupervised learning utilizing the results of an existing VLM. The LoRA learning for this language model (230) will be described in more detail below.

[0048] FIG. 3 is a diagram illustrating an example of a model for object recognition and / or image classification according to an embodiment of the present invention. The model (210) for object recognition and / or image classification may include n object recognition models and m image classification models. In this case, n may be 0 or a natural number, m may be 0 or a natural number, and the sum of n and m may be 1 or greater. In other words, the image caption generation system (200) may utilize only one or more object recognition models or only one or more image classification models, depending on the embodiment. In addition, the image caption generation system (200) may utilize one or more object recognition models and one or more image classification models simultaneously.

[0049] An image (240) can be input into each of n object recognition models and m image classification models included in a model (210) for object recognition and / or image classification, and the results of each of the n object recognition models and m image classification models can be input into a prompt processing unit (220). The prompt processing unit (220) can generate a prompt for a language model (230) using the results of each of the n object recognition models and m image classification models.

[0050] Meanwhile, object recognition models and image classification models differ in the classes they support. For example, an object recognition model can support approximately 80 classes, while an image classification model can support approximately 1,000 classes. Therefore, it is necessary to consider the difference in the level of detailed support depending on the application. Accordingly, the prompt processing unit (220) can use information about predefined groups to map and link the labels (classes) output by each model to group different classes. For example, assume that labels such as {Event, Leisure, Surfboard, Vacation, Surfing, Sports} are output through image classification for a specific image. Furthermore, assume that labels such as {Surfboard, Person} are output through object recognition for the same image. In this case, the prompt processing unit (220) can map and link the labels "Surfboard" and "Surfing" output through image classification and "Surfboard" output through object recognition, group them into the same group, and use them to generate prompts. Among the labels output through image classification, labels such as "Event," "Leisure," "Vacation," and "Sports," which describe images regardless of the object, can be utilized individually. As another example, the labels "Maltese," "Dalmatian," and "Pug" output through image classification and the label "Dog" output through object recognition can be mapped and linked together to form the same group and utilized for prompt generation.

[0051] In addition, since the image caption generation system (200) generates input for the language model (230) from both the results of image classification and object recognition, the prompt processing unit (220) can optimize the input format by utilizing prompt engineering. For example, the image caption generation system (200) can optimize the prompt by comparing the image caption generated using the generated prompt with the image caption generated using the VLM. Here, the VLM can be implemented on the server side. For example, the image caption generation system (200) can transmit the image (240) to the server (260), and receive the image caption obtained by inputting the image (240) into the VLM in the server (260) from the server (260). Thereafter, the image caption generation system (200) can optimize the prompt by comparing the image caption generated using the generated prompt with the image caption transmitted from the server (260). Communication between the image caption generation system (200) and the server (260) may be accomplished through data transmission and reception via a network (160) between a computer device (100) implementing the image caption generation system (200) and a device implementing the server (260). As another example, the image caption generation system (200) may perform additional work to improve prompts by utilizing a language model (230).

[0052] Table 1 below shows examples of prompts before and after applying prompt engineering.

[0053] AS-IS"Person(coordinates), Surfboard(coordinates), Fun, Leisure, Sufboard. Please generate the caption of the Image"TO-BE"The followings are results from object detection and image labeling.```Object detection: Person(coordinates), Surfboard(coordinates)Image Labeling: Fun, Leisure, Sufboard```By using above labels and position coordination, describe the given image within 3 sentences."

[0054] FIGS. 4 and 5 are drawings illustrating examples of a process for generating an image caption according to one embodiment of the present invention.

[0055] FIG. 4 illustrates an object recognition result (420) and an image classification result (430) for an image (410), respectively. The object recognition result (420) may include information about a label and a position of an object corresponding to the label within the image (410). In the present embodiment, an example is described in which the coordinates of the object are represented by the x-axis minimum value "xmin", the y-axis minimum value "ymin", the x-axis maximum value "xmax", and the y-axis maximum value "ymax" of a rectangle representing the area of ​​the object as the information about the position, but the information about the position of the object may be configured in various ways, such as the coordinates of the diagonal corners of the rectangle. The image classification result (430) may include information about a label and the probability of each label. The number in parentheses of each label in the image classification result (430) of FIG. 4 may indicate an identifier (ID) of the label. The label of the object recognition result (420) may also have a corresponding identifier.

[0056] Fig. 5 shows an example of a merge result (510) for each label of an object recognition result (420) and an image classification result (430). The merge result (510) of Fig. 5 shows an example in which the labels "Surfboard" and "Surfing" of the image classification result (430) are integrated into the label (coordinates) "surfboard (1303, 1084, 4509, 1244)" of the object recognition result (420). In addition, Fig. 5 shows an example (520) in which a simple prompt (Question 1) is generated based on the merge result (510) and inputted into a language model (e.g., a language model (230)), thereby obtaining an output (Answer 1) of the language model. In addition, Fig. 5 shows an example (530) in which a complex prompt (Question 2) is generated based on the merge result (510) and inputted into a language model, thereby obtaining an output (Answer 2) of the language model. For example, a prompt can be generated through the prompt processing unit (220) described above. As shown in the embodiment of Fig. 5, it can be confirmed that the output of the language model can also vary depending on the prompt.

[0057] FIGS. 6 and 7 are drawings illustrating another example of the process of generating an image caption according to one embodiment of the present invention.

[0058] Fig. 6 shows an object recognition result (620) and an image classification result (630) for an image (610), respectively. The object recognition result (620) may include information about a label and a position of an object corresponding to the label within the image (610). In this embodiment, an example is described that includes coordinates of an object represented by an x-axis minimum value "xmin", a y-axis minimum value "ymin", an x-axis maximum value "xmax", and a y-axis maximum value "ymax" of a rectangle representing an area of ​​the object as information about the position. The image classification result (630) may include information about a label and the probability of each label.

[0059] Fig. 7 shows an example of a merge result (710) for each label of the object recognition result (620) and the image classification result (630). The merge result (710) of Fig. 7 shows an example in which the labels "Bicycle" and "Race" of the image classification result (630) were removed because they were judged to be inaccurate classifications based on the labels (coordinates) "Person (183,207,207,255)" and "Umbrella (175,193,212,208)" of the object recognition result (620). In addition, Fig. 7 shows an example in which a simple prompt (Question 1) was generated based on the merge result (710) and inputted into a language model, thereby obtaining the output (Answer 1) of the language model. In addition, Fig. 7 shows an example in which a complex prompt (Question 2) was generated based on the merge result (710) and inputted into a language model, thereby obtaining the output (Answer 2) of the language model. For example, a prompt can be generated through the prompt processing unit (220) described above. As shown in the embodiment of Fig. 7, it can be confirmed that the output of the language model can also vary depending on the prompt.

[0060] FIG. 8 is a diagram illustrating an example of a LoRA model according to an embodiment of the present invention. Existing language models are not created for specific purposes, but rather are pre-trained to generate generally well-expressed answers to user queries. Therefore, fine-tuning is required to optimize the language model so that it can generate higher-quality image captions for the same prompt, even for image caption generation. However, fine-tuning a language model is costly and time-consuming. Therefore, in this embodiment, image captions can be generated by fine-tuning the language model through LoRA model training. For example, the language model can be trained as an optimized language model for generating image captions using textual information about labels and object locations. Since the language model itself is too large to train, the LoRA model training can divide the language model into two smaller sub-models. The embodiment of FIG. 8 illustrates an example of deriving results using a LoRA model divided into two models of weights A and B instead of the pre-trained weights W (= A Х B) of the existing language model. At this time, since the size of models A and B is very small compared to the existing language model (for example, 10000 = 100 Х 100), it is easy to create a LoRA model with fine-tuned language model even on-device. In other words, while the learning speed takes exponentially longer as the size increases, since A and B are very small compared to W, learning is possible in a short time.

[0061] Furthermore, these LoRA models can be fine-tuned to suit specific needs. For example, separate LoRA models can be created for different languages. In this case, by using label identifiers (e.g., the identifier "386" for "Fun" and the identifier "239" for "Leisure") instead of label text (e.g., the English "Fun" or "Leisure"), it becomes possible to create LoRA models that generate image captions appropriate for each language. Furthermore, by only changing the LoRA model for the language model in real time, it becomes possible to generate image captions appropriate for different languages ​​or purposes, without having to change the model (210) for object recognition and / or image classification and the prompt processing unit (220).

[0062] Additionally, the training data can utilize the image caption generation results of the VLM. In this case, the image caption generation results of the VLM can be generated on the server side (e.g., server (260)) and transmitted to the client side of the on-device device. In this case, the on-device device can fine-tune the language model based on LoRA model training using the transmitted training data to generate a LoRA model.

[0063] FIG. 9 is a flowchart illustrating an example of a method for generating an image caption according to an embodiment of the present invention. The method for generating an image caption according to the present embodiment may be performed by at least one computer device (100). At this time, the processor (120) of the computer device (100) may be implemented to execute control instructions according to the code of the operating system included in the memory (110) or the code of at least one computer program. Here, the processor (120) may control the computer device (100) so that the computer device (100) performs steps (910 to 950) included in the method of FIG. 9 according to the control instructions provided by the code stored in the computer device (100).

[0064] In step (910), the computer device (100) can generate a result of image classification or object recognition for the input image. For example, the computer device (100) can generate the output of one or more image classification models for image classification as a result, generate the output of one or more object recognition models for object recognition as a result, or generate the output of one or more image classification models and the output of one or more object recognition models as a result. As described above, it has been explained that the results as outputs of n object recognition models and m image classification models can be utilized.

[0065] In step (920), the computer device (100) may generate a prompt for the language model based on the generated result. Consider the case where, in step (910), the computer device (100) generates a first output of one or more image classification models and a second output of one or more object recognition models as results. In this case, in step (920), the computer device (100) may merge the first output and the second output into a single result and generate a prompt using the merged result. As previously described, groups that can group different classes (or labels) may be predefined, and classes (or labels) may be mapped and linked according to these groups.

[0066] Here, the language model may be included in a computer program for generating an image caption and loaded onto a computer device (100), but may also include a language model implemented on-device in the computer device (100) or the operating system of the computer device (100). When using a language model implemented on-device, the computer program may include a function for calling and utilizing the language model. Since the technology for an application to utilize a function pre-implemented in the computer device (100) is already well known, a detailed description thereof will be omitted.

[0067] In some embodiments, the computer device (100) may optimize the prompt at step (920). For example, the computer device (100) may optimize the prompt by inputting the result generated at step (920) into a language model and generating the prompt using the output of the language model.

[0068] In step (930), the computer device (100) may input the generated prompt into a language model to generate an image caption for the image. The computer device (100) may utilize the language model as is, but depending on the embodiment, the language model may be fine-tuned through LoRA learning to generate and utilize a LoRA model. For example, the computer device (100) may generate a LoRA model by fine-tuning the language model according to a predetermined purpose through LoRA learning on the language model. The predetermined purpose may mainly be generating an appropriate image caption for the image through a prompt generated through the results of image classification and / or object recognition for the image. In addition, the predetermined purpose may include a predetermined language. For example, to generate an image caption including a Japanese sentence, the language model may be fine-tuned through LoRA learning to generate an image caption suitable for Japanese.

[0069] Additionally, the computer device (100) can input an image into the VLM and utilize the first image generation result as training data for LoRA learning to fine-tune the language model. As previously described, since the VLM is difficult to implement on-device, the VLM may be implemented on a server (e.g., server (260)). In this case, the training data may be generated on the server and transmitted to the computer device (100).

[0070] Once the LoRA model is generated, the computer device (100) can generate an image caption for the image by inputting the generated prompt into a Large Language Model (LLM) to which the generated LoRA model is applied.

[0071] In another embodiment, the computer device (100) can utilize VLM to optimize prompts and generate additional image captions using the optimized prompts. To this end, subsequent steps (940) and (950) may be performed. If VLM is not utilized to optimize prompts, steps (940) and (950) may be omitted.

[0072] In step (940), the computer device (100) can compare a first image caption generated by inputting an image into the VLM with a second image caption generated by inputting a prompt into the language model to modify the prompt. As previously described, since the VLM is difficult to implement on-device, the VLM may be implemented in a server (e.g., server (260)). In this case, the computer device (100) can transmit the image to the server. At this time, the server can input the image into the VLM implemented in the server to generate a first image caption and transmit the generated first image caption to the computer device (100). In this case, the computer device (100) can optimize the prompt by comparing the received first image caption with a second image caption generated by inputting the prompt into the language model.

[0073] At step (950), the computer device (100) may input the modified prompt into a language model to generate additional image captions for the images.

[0074] In this way, according to embodiments of the present invention, a method and system for generating image captions using image classification and / or object recognition can be provided.

[0075] The systems or devices described above may be implemented as hardware components, or a combination of hardware components and software components. For example, the devices and components described in the embodiments may be implemented using one or more general-purpose computers or special-purpose computers, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding to them. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device is sometimes described as being used alone; however, one of ordinary skill in the art will recognize that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, a processing unit may include multiple processors, or a processor and a controller. Other processing configurations, such as parallel processors, are also possible.

[0076] Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing device to perform a desired operation or may independently or collectively command the processing device. The software and / or data may be embodied in any type of machine, component, physical device, virtual equipment, computer storage medium, or device for interpretation by the processing device or for providing instructions or data to the processing device. The software may also be distributed over networked computer systems and stored or executed in a distributed manner. The software and data may be stored on one or more computer-readable recording media.

[0077] The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program commands, data files, data structures, etc., singly or in combination. The medium may continuously store a computer-executable program, or may temporarily store it for execution or download. In addition, the medium may be various recording means or storage means in the form of a single or multiple hardware combinations, and is not limited to a medium directly connected to a computer system, but may also be distributed over a network. Examples of the medium may include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and those configured to store program commands, including ROM, RAM, and flash memory. In addition, examples of other media may include recording media or storage media managed by app stores that distribute applications, sites that supply or distribute various software, servers, etc. Examples of program instructions include machine language code, such as that produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc.

[0078] Although the embodiments described above have been described by way of limited examples and drawings, those skilled in the art will appreciate that various modifications and variations can be made based on the above teachings. For example, appropriate results can still be achieved even if the described techniques are performed in a different order than described, and / or components of the described systems, structures, devices, circuits, etc. are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents.

[0079] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims set forth below.

Claims

1. A computer program stored in a computer-readable recording medium for executing a method for generating image captions on a computer device by being combined with a computer device, The above image caption generation method is, A step of generating a result of image classification or object recognition for an input image; A step of generating a prompt for a language model based on the generated results; and A step of generating an image caption for the image by inputting the generated prompt into the language model. including A computer program characterized by .

2. In paragraph 1, The steps for generating the above results are: Generating the output of one or more image classification models for the image classification as the result, or generating the output of one or more object recognition models for the object recognition as the result, or generating the output of one or more image classification models and the output of one or more object recognition models as the result. A computer program characterized by .

3. In paragraph 1, The steps for generating the above results are: Generating a first output of one or more image classification models and a second output of one or more object recognition models as the results, The steps to create the above prompt are: Merging the first output and the second output into one result, and generating the prompt using the merged result. A computer program characterized by .

4. In paragraph 1, The steps to create the above prompt are: Inputting the generated result into the language model and generating the prompt using the output of the language model A computer program characterized by .

5. In paragraph 1, The above image caption generation method is, A step of comparing a first image caption generated by inputting the image into a VLM (Vision Language Model) and a second image caption generated by inputting the prompt into the language model to modify the prompt; and A step of generating an additional image caption for the image by inputting the above modified prompt into the above language model. Including more A computer program characterized by .

6. In paragraph 5, The steps to modify the above prompt are: A step of transmitting the above image to a server; A step of receiving the first image caption generated by inputting the image into the VLM (Vision Language Model) implemented in the server from the server; and A step of modifying the prompt by comparing the first image caption and the second image caption. including A computer program characterized by .

7. In paragraph 1, The above language model includes a language model implemented in an on-device form in the computer device or the operating system of the computer device. A computer program characterized by .

8. In paragraph 1, The steps for generating the above image caption are: A step of generating a LoRA model by fine-tuning the language model according to a predetermined purpose through LoRA (Low-Rank Adaptation) learning for the language model; and A step of generating an image caption for the image by inputting the generated prompt into a Large Language Model (LLM) to which the generated LoRA model is applied. including A computer program characterized by .

9. In paragraph 8, The above preset purpose includes the preset language. A computer program characterized by .

10. In paragraph 8, The steps for creating the above LoRA model are: Fine-tuning the language model by using the first image generation result generated by inputting the image to the VLM as learning data for the LoRA learning. A computer program characterized by .

11. In paragraph 10, The above VLM is implemented on the server, The above learning data is generated and transmitted using the VLM on the server. A computer program characterized by .

12. A method for generating an image caption of a computer device including at least one processor, A step of generating a result of image classification or object recognition for an input image by at least one processor; generating a prompt for a language model based on the generated result by at least one processor; and A step of generating an image caption for the image by inputting the generated prompt into the language model by the at least one processor. A method for generating image captions that include:

13. In paragraph 12, The steps for generating the above results are: Generating the output of one or more image classification models for the image classification as the result, or generating the output of one or more object recognition models for the object recognition as the result, or generating the output of one or more image classification models and the output of one or more object recognition models as the result. A method for generating image captions featuring:

14. In paragraph 12, The steps for generating the above results are: Generating a first output of one or more image classification models and a second output of one or more object recognition models as the results, The steps to create the above prompt are: Merging the first output and the second output into one result, and generating the prompt using the merged result. A method for generating image captions featuring:

15. In paragraph 12, The steps to create the above prompt are: Inputting the generated result into the language model and generating the prompt using the output of the language model A method for generating image captions featuring:

16. In paragraph 12, A step of comparing a first image caption generated by inputting the image into a VLM (Vision Language Model) and a second image caption generated by inputting the prompt into the language model to modify the prompt; and A step of generating an additional image caption for the image by inputting the above modified prompt into the above language model. A method for generating image captions that include more.

17. In paragraph 12, The above language model includes a language model implemented in an on-device form in the computer device or the operating system of the computer device. A method for generating image captions featuring:

18. In paragraph 12, The steps for generating the above image caption are: A step of generating a LoRA model by fine-tuning the language model according to a predetermined purpose through LoRA (Low-Rank Adaptation) learning for the language model; and A step of generating an image caption for the image by inputting the generated prompt into a Large Language Model (LLM) to which the generated LoRA model is applied. A method for generating an image caption, characterized in that it includes:

19. A computer-readable recording medium having recorded thereon a computer program for executing the method of any one of claims 12 to 18 on a computer device.

Images