Device and method for generating image caption

The method addresses the limitation of focusing on main objects by searching for similar images with LVLM to enhance captioning accuracy, detailing minor features effectively.

KR103044716B1Active Publication Date: 2026-11-06UNIV OF SEOUL IND COOP FOUND
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
KR1020240065078
Authority / Receiving Office
KR · KR
Patent Type
Patents
Current Assignee / Owner
Filing Date
2024-05-20
Publication Date
2026-11-06
Estimated Expiration
2044-05-20

AI Technical Summary

Technical Problem

Existing image caption generation methods focus solely on the main object in a target image, neglecting other features, resulting in inadequate descriptions.

Method used

An image caption generation method that searches for similar images with high similarity to the target image, using a Large Vision-Language Model (LVLM) to determine differences and generate captions based on these similarities, creating a virtuous cycle of improved captioning accuracy.

Benefits of technology

Generates captions that more accurately reflect the features of a target image by focusing on detailed and minor parts, achieving high accuracy without large data sets or additional training, suitable for low-power devices.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 112024054038112-PAT00004_ABST
    Figure 112024054038112-PAT00004_ABST
Patent Text Reader

Abstract

The present invention relates to an image caption generation device and method capable of searching for similar images with high similarity to the image caption of a target image, and adding a caption for the target image based on the difference between the caption of the searched similar image and the target image. An image caption generation method performed by the image caption generation device described above may include: (a) a step of receiving a target image; (b) a step of determining a similar image for the target image based on the target image and a target caption which is an image caption for the target image; (c) a step of generating a reinforcement caption for the target image based on the similar caption which is an image caption for the similar image; (d) a step of adding the reinforcement caption to the target caption; and (e) a step of repeating steps (b) through (d).
Need to check novelty before this filing date? Find Prior Art

Description

Technology Field

[0001] The present invention relates to an image caption generation device and method.

[0002] Specifically, the present invention relates to an image caption generation device and method capable of searching for similar images with high similarity to the image caption of a target image, and adding a caption for the target image based on the difference between the caption of the searched similar image and the target image. Background Technology

[0004] Caption refers to text data that describes the characteristics of visual content, such as images and videos. Captions serve the role of explaining the content or providing additional information.

[0005] Generally, to generate such image captions, a target image is input, and a process of extracting features from that image is performed. However, if captions are generated by referencing only the target image, a problem arises in that the focus becomes solely on the main objects contained within the image. In other words, the issue occurs where the caption is generated by focusing only on the object with the highest occupancy rate within the target image.

[0006] In this case, features of objects other than the main object included in the target image are not included in the caption, and consequently, the caption cannot adequately describe the features of the target image. The problem to be solved

[0008] The problem that the present invention aims to solve is to provide an image caption generation device and method capable of generating a caption that more accurately reflects the features of a target image by searching for a similar image with high similarity to the caption of a target image and adding a caption for the target image based on the difference between the caption of the searched similar image and the target image.

[0009] Specifically, the problem that the present invention aims to solve is to provide an image caption generation device and method that can generate a caption for a target image more simply and quickly by determining the difference between the caption of the similar image and the visual features of the target image using a Large Vision-Language Model (LVLM) when a similar image is searched.

[0010] Furthermore, the problem that the present invention aims to solve is to provide an image caption generation device and method with a virtuous cycle structure in which, by repeating the process of searching for similar images and the process of generating captions through comparison with similar images, the better the caption used to search for similar images describes the target image, the more similar images with higher similarity are searched, and the more differences from the target image are asked after searching for similar images, the more detailed and minor parts are focused on to generate captions.

[0011] The objects of the present invention are not limited to those mentioned above, and other unmentioned objects and advantages of the present invention may be understood from the following description and will be more clearly understood by the embodiments of the present invention. Furthermore, it will be readily apparent that the objects and advantages of the present invention can be realized by the means and combinations thereof set forth in the claims. means of solving the problem

[0013] An image caption generation method performed by an image caption generation device according to some embodiments of the present invention may include: (a) a step of receiving a target image; (b) a step of determining a similar image for the target image based on the target image and a target caption which is an image caption for the target image; (c) a step of generating a reinforcement caption for the target image based on the similar caption which is an image caption for the similar image; (d) a step of adding the reinforcement caption to the target caption; and (e) a step of repeating steps (b) through (d).

[0014] Additionally, the above step (b) may include a step of identifying similar images for a target image based on a predefined identification algorithm.

[0015] In addition, the identification algorithm includes an algorithm based on a Vision-Language Model (VLM), and the VLM may include a Contrastive Language-Image Pre-training (CLIP) model.

[0016] Additionally, the identification algorithm includes a text embedding unit and an image embedding unit, and the step of identifying a similar image for a target image based on the identification algorithm may include the step of generating a text embedding by inputting the target caption into the text embedding unit, and the step of identifying the similar image based on the similarity between the text embedding and a plurality of comparison images stored in advance.

[0017] Additionally, the step of identifying the similar image based on the similarity between the text embedding and a plurality of pre-stored comparison images may include the step of generating a plurality of image embeddings by inputting each of the plurality of comparison images into the image embedding unit, and the step of identifying the similar image based on the cosine similarity between the text embedding and the image embeddings.

[0018] Additionally, the above step (c) may include the step of determining a predetermined number of selected captions among the similar captions, and the step of generating the reinforcement caption based on the selected captions.

[0019] Additionally, the step of determining a predetermined number of selected captions among the similar captions may include the step of randomly extracting two similar captions from a plurality of similar captions and determining them as the selected captions.

[0020] Additionally, the step of generating the reinforcement caption based on the above-mentioned selection caption may include the step of generating the reinforcement caption based on the target image, the above-mentioned selection caption, and a predefined generation algorithm.

[0021] In addition, the above generation algorithm may include an algorithm based on the Large Vision-Language Model (LVLM).

[0022] Additionally, the step of generating the reinforcement caption based on the target image, the selected caption, and a predefined generation algorithm may include the step of generating the reinforcement caption by inputting the target image, the selected caption, and request data requesting visual distinction between the target image and the selected caption into the generation algorithm. Effects of the invention

[0024] An image caption generation device and method according to some embodiments of the present invention have a significant effect of generating a caption that more accurately reflects the features of a target image by searching for a similar image that has high similarity to the caption of a target image and adding a caption for the target image based on the difference between the caption of the searched similar image and the target image. At this time, when a similar image is searched, the image caption generation device and method according to some embodiments of the present invention can generate a caption for a target image more simply and quickly by determining the difference between the caption of the similar image and the visual features of the target image using LVLM.

[0025] In addition, the image caption generation device and method according to some embodiments of the present invention can achieve a virtuous cycle structure in which, by repeating the process of searching for similar images and the process of generating captions through comparison with similar images, the more well the caption used to search for similar images describes the target image, the more similar images with higher similarity are searched, and the more differences from the target image are asked after searching for similar images with high similarity, the more detailed and minor parts are focused on to generate captions.

[0026] Furthermore, the image caption generation device and method according to some embodiments of the present invention can capture and represent detailed image information without additional training or large data sets. That is, through the image caption generation device and method according to some embodiments of the present invention, efficient and effective image captioning can be achieved without large data sets or separate training, and accordingly, high performance can be maintained even in environments with low-power devices or limited computational resources.

[0027] Furthermore, the image caption generation device and method according to some embodiments of the present invention can enable the utilization of Large-Language Models (LMs) that receive only text data as input in the field of vision by generating detailed and differentiated image captions. That is, the image caption generation device and method according to some embodiments of the present invention converts visual data into natural language to significantly improve the accuracy and detail of image understanding, and by reducing the gap between the visual and language models, enables the model to process visual and linguistic data integrally.

[0028] In addition, image captions generated according to the image caption generation device and method according to some embodiments of the present invention have the potential to be utilized in fields such as image-based information retrieval and social media content creation, and can be expanded in various ways such as automatic description of visual content. Furthermore, by providing more specific and differentiated descriptions of images, they have the potential to be applied in various industrial fields.

[0029] In addition to the above, the specific effects of the present invention are described together with the specific details for implementing the invention below. Brief explanation of the drawing

[0031] FIG. 1 illustrates an image caption generation system according to some embodiments of the present invention. FIG. 2 is a block diagram of an image caption generation device according to some embodiments of the present invention. FIG. 3 is a diagram illustrating the neural network structure of a deep learning algorithm according to some embodiments of the present invention. FIG. 4 is a flowchart of an image caption generation process according to some embodiments of the present invention. FIG. 5 is a conceptual diagram illustrating an image caption generation process according to some embodiments of the present invention. Figure 6 is a detailed flowchart of the step for determining similar images of Figure 4. Figure 7 is a detailed flowchart of the steps for generating the reinforcement caption of Figure 4. FIG. 8 is a diagram illustrating a hardware implementation of an image caption generation device according to some embodiments of the present invention. Specific details for implementing the invention

[0032] Terms and words used in this specification and claims shall not be interpreted as being limited to their general or dictionary meanings. In accordance with the principle that an inventor may define the concept of a term or word to best describe their invention, they shall be interpreted in a meaning and concept consistent with the technical spirit of the invention. Furthermore, since the embodiments described in this specification and the configurations illustrated in the drawings are merely one embodiment of the invention and do not represent the entire technical spirit of the invention, it should be understood that various equivalents, modifications, and applicable examples capable of replacing them may exist at the time of filing this application.

[0033] The terms first, second, A, B, etc., as used in this specification and claims may be used to describe various components, but said components should not be limited by said terms. These terms are used solely for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be named the second component, and similarly, the second component may be named the first component. The term "and / or" includes a combination of a plurality of related described items or any of a plurality of related described items.

[0034] The terms used in this specification and claims are used merely to describe specific embodiments and are not intended to limit the invention. The singular expression includes the plural expression unless the context clearly indicates otherwise. In this application, terms such as "comprising" or "having" should be understood as not precluding the existence or addition of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification.

[0035] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by those skilled in the art to which this invention pertains.

[0036] Terms such as those defined in commonly used dictionaries should be interpreted as having meanings consistent with their meanings in the context of the relevant technology, and should not be interpreted in an ideal or overly formal sense unless explicitly defined in this application.

[0037] In addition, each component, process, procedure, or method included in each embodiment of the present invention may be shared within a scope that is not technically contradictory to one another.

[0038] Hereinafter, with reference to FIGS. 1 to 8, an image caption generation device and method according to some embodiments of the present invention and a system including the same will be described in detail.

[0040] FIG. 1 illustrates an image caption generation system according to some embodiments of the present invention.

[0041] Referring to FIG. 1, an image caption generation system (1) according to some embodiments of the present invention may include an external database (100), a user terminal (200), an image caption generation device (300), and a communication network (400).

[0042] The external database (100) may be a database that stores, manages, and / or transmits multiple image data that form the basis of the image caption generation operation.

[0043] For example, an external database (100) may store multiple image data and / or image captions for each image data. In this case, an image caption refers to text data that describes the characteristics of the image data. In other words, the image caption may include text data that performs the role of describing the content of the image data or providing additional information.

[0044] An external database (100) may transmit image data and / or image captions for said image data to an image caption generation device (300). At this time, the image data transmitted by the external database (100) to the image caption generation device (300) may include a target image that is the subject of image caption generation and / or a comparison image used for said image captioning.

[0045] Meanwhile, the external database (100) may be in the form of various types of electronic devices such as computers, laptop PCs, mobile devices, and wearable devices, workstations, data centers, internet data centers (IDCs), direct attached storage (DAS) systems, storage area network (SAN) systems, network attached storage (NAS) systems, and redundant array of inexpensive disks, or redundant array of independent disks) systems, but the embodiments of the present invention are not limited thereto.

[0046] The user terminal (200) is a user terminal that requests an image caption generation task from the image caption generation device (300).

[0047] In some examples, a user terminal (200) may transmit image data and / or an image caption for said image data to an image caption generating device (300) for an image caption generating operation. At this time, the image data transmitted by the user terminal (200) to the image caption generating device (300) may include a target image that is the subject of image caption generation and / or a comparison image used for said image captioning operation.

[0048] Meanwhile, the user terminal (200) may be in the form of various types of electronic devices such as smartphones, computers, laptop PCs, and wearable devices, workstations, data centers, internet data centers (IDCs), direct attached storage (DAS) systems, storage area network (SAN) systems, network attached storage (NAS) systems, and redundant array of inexpensive disks, or redundant array of independent disks) systems, but the embodiments of the present invention are not limited thereto.

[0049] The image caption generation device (300) means an electronic device or server that performs the image caption generation task.

[0050] In some examples, an image caption generation device (300) may receive image data to be the target of image caption generation and / or image captions for each image data from an external database (100) or a user terminal (200), and perform an image caption generation operation using the received data. At this time, the image data received by the image caption generation device (300) may include a target image to be the target of image caption generation and a comparison image used for the image caption operation.

[0051] However, embodiments of the present invention are not limited thereto, and in some embodiments of the present invention, an external database (100) and a user terminal (200) may be omitted in an image caption generation system (1). In this case, the image caption generation device (300) may perform an image caption generation operation without performing data exchange with the external database (100) and the user terminal (200). For example, the image caption generation device (300) may perform an image caption generation operation using image data (target image, comparison image) stored in a database module (320 in FIG. 2) included in the image caption generation device (300) and image captions for each image data. However, for the convenience of the following explanation, it will be described by assuming a case where the image caption generation device (300) receives and uses multiple image data and image captions from the external database (100) and the user terminal (200).

[0052] A detailed description of the image caption generation operation performed by the image caption generation device (300) will be provided later.

[0053] Meanwhile, the image caption generating device (300) may be in the form of a workstation, a data center, an internet data center (IDC), a direct attached storage (DAS) system, a storage area network (SAN) system, a network attached storage (NAS) system, and a redundant array of inexpensive disks, or a redundant array of independent disks (RAID) system, but the embodiments of the present invention are not limited thereto.

[0054] The communication network (400) serves to connect the external database (100), the user terminal (200), and the image caption generation device (300). That is, the communication network (400) refers to a communication network that provides a connection path so that the image caption generation device (300) can transmit and receive data from the external database (100) and the user terminal (200). The communication network (400) may include wired networks such as LANs (Local Area Networks), WANs (Wide Area Networks), MANs (MetRoFolitan Area Networks), and ISDNs (Integrated Service Digital Networks), or wireless networks such as wireless LANs, CDMA, Bluetooth, and satellite communication, but the scope of the present invention is not limited thereto.

[0055] Hereinafter, the operation of the image caption generation device (300) will be explained in more detail with reference to FIG. 2.

[0057] FIG. 2 is a block diagram of an image caption generation device according to some embodiments of the present invention.

[0058] Referring to FIGS. 1 and 2, the image caption generation device (300) may include a communication module (310), a database module (320), and an execution module (330). However, embodiments of the present invention are not limited thereto, and in some embodiments of the image caption generation device (300) according to the present invention, some components may be omitted or other components not shown may be added. For example, the communication module (310) or the database module (320) may be omitted in the image caption generation device (300).

[0059] The communication module (310) can receive a target image (hereinafter referred to as "TI"). In other words, the communication module (310) can receive a target image (TI) that is the target for generating an image caption. At this time, the communication module (310) can receive the target image (TI) from an external database (100) or a user terminal (200), but the embodiment of the present invention is not limited thereto, and as described above, the target image (TI) may be stored in advance in a database module (320) within the image caption generation device (300). Hereinafter, for convenience of explanation, the case in which the communication module (310) receives the target image (TI) from an external database (100) or a user terminal (200) will be described.

[0060] The communication module (310) can transmit the received target image (TI) to other components within the image caption generation device (300). For example, the communication module (310) can transmit the target image (TI) to the execution module (330), etc., but the present invention is not limited thereto. The communication module (310) can perform data exchange between the external database (100) and user terminal (200) and the image caption generation device (300) through the communication network (400).

[0061] The database module (320) may be a module that stores and manages various data required for the image caption generation operation of the image caption generation device (300).

[0062] However, the embodiments of the present invention are not limited thereto, and the database module (320) in the image caption generation device (300) may be omitted. In this case, the image caption generation device (300) may receive and use various data required for the image caption generation operation from an external database (100) or a user terminal (200). Hereinafter, for convenience of explanation, the description will assume that the image caption generation device (300) includes a database module (320).

[0063] For example, the database module (320) can store and manage multiple comparison images (hereinafter referred to as "CI"). For instance, the database module (320) can receive multiple comparison images (CI) from an external database (100) or a user terminal (200) through the communication module (310), and can store and manage the received comparison images (CI).

[0064] As another example, the database module (320) can store and manage a deep learning algorithm (Deep Learning Algorithm, hereinafter referred to as "DA") used for image caption generation tasks.

[0065] The deep learning algorithm (DA) may include an identification algorithm (IA in FIG. 5), a generation algorithm (GA in FIG. 5), etc. In this case, the identification algorithm (IA in FIG. 5) may be used in the similar image identification process of the execution module (330) described later, and the generation algorithm (GA in FIG. 5) may be used in the reinforcement caption generation process of the execution module (330) described later. A detailed description of each algorithm will be provided later.

[0066] These deep learning algorithms (DA) can be trained based on machine learning.

[0067] To explain in more detail, deep learning, a type of machine learning, involves learning by descending to deep levels in multiple stages based on data. In other words, deep learning represents a set of machine learning algorithms that extract key data from multiple datasets by progressively increasing the levels.

[0068] As a few examples, deep learning algorithms (DA) can utilize various known deep learning structures. For example, deep learning algorithms (DA) can utilize structures such as Convolutional Neural Network (CNN), Recurrent Neural Network (RNN), Deep Belief Network (DBN), Graph Neural Network (GNN), Generative Adversarial Network (GAN), Transformer, and Autoencoder.

[0069] Specifically, a Convolutional Neural Network (CNN) is a model that mimics the function of the human brain, based on the assumption that when humans recognize an object, they extract its basic features, perform complex calculations within the brain, and then recognize the object based on the results. CNNs may include well-known structures such as LeNet, AlexNet, VGGNet, GoogleNet, and ResNet, but are not limited to them.

[0070] Recurrent Neural Networks (RNNs) are widely used in natural language processing and are an effective structure for processing time-series data that changes over time; they can be constructed by stacking layers at every moment.

[0071] A Deep Belief Network (DBN) is a deep learning structure constructed by stacking Restricted Boltzmann Machines (RBMs), a deep learning technique, in multiple layers. When the Restricted Boltzmann Machine (RBM) training is repeated until a certain number of layers are reached, a Deep Belief Network (DBN) with that number of layers can be constructed.

[0072] A Graphic Neural Network (GNN) represents an artificial neural network structure implemented by deriving similarities and feature points between modeling data using modeling data modeled based on data mapped between specific parameters.

[0073] A Generative Adversarial Network (GAN) represents an artificial neural network structure that uses a generative neural network and a discriminative neural network to generate new data in a form similar to input data. GANs may include known DCGAN (Deep Convolutional GAN), CGAN (Conditional GAN), WGAN (Wasserstein GAN), StyleGAN (Style-Based GAN), CycleGAN, etc., but embodiments of the present invention are not limited thereto.

[0074] The Transformer is an artificial neural network with an encoder-decoder structure utilizing attention, capable of grasping the overall meaning between input and output sequences. By employing an attention mechanism, the Transformer ensures that every element of the input sequence influences the output sequence, allowing both the encoder and decoder to consider the entire sequence. The Transformer can use natural language and time-series data, as well as patched images, as input.

[0075] An autoencoder is a deep learning architecture that performs the role of extracting and reconstructing data features. Typically, an autoencoder includes an encoder that compresses input values ​​and a decoder that restores the compressed data. The encoder transforms input values ​​into low-dimensional latent representations, while the decoder restores the latent representations to the same dimension as the input values. In this process, both the encoder and decoder can be composed of Multilayer Perceptrons (MLPs). When training an autoencoder, input data is used, and weights and biases are trained to minimize the difference between the output and input values. An autoencoder trained in this way can effectively extract features from input data and restore noisy input data. Autoencoders are primarily utilized in fields such as data compression, dimensionality reduction, noise removal, and data generation; they can also be applied in areas such as image recognition, natural language processing, and speech recognition.

[0076] Meanwhile, the training of artificial neural networks in deep learning algorithms (DA) can be achieved by adjusting the weights of the connections between nodes (and, if necessary, adjusting the bias values) to produce a desired output for a given input. Additionally, artificial neural networks can continuously update weight values ​​through learning. Furthermore, methods such as backpropagation can be used for the training of artificial neural networks.

[0077] At this time, machine learning methods for artificial neural networks, such as unsupervised learning, semi-supervised learning, and supervised learning, may be used. In addition, the deep learning algorithm (DA) can be controlled to automatically update the artificial neural network structure to output analysis data after learning, depending on the settings.

[0078] Hereinafter, with reference to FIG. 3, the structure of a neural network of a deep learning algorithm (DA) according to some embodiments of the present invention will be described.

[0080] FIG. 3 is a diagram illustrating the neural network structure of a deep learning algorithm according to some embodiments of the present invention.

[0081] Referring to FIGS. 2 and FIGS. 3, a deep learning algorithm (DA) stored in a database module (320) according to some embodiments of the present invention may include an input layer, an output layer, and M hidden layers placed between the input layer and the output layer.

[0082] At this time, as described above, the deep learning algorithm (DA) may include an identification algorithm (IA in FIG. 5), a generation algorithm (GA in FIG. 5), etc. At this time, the identification algorithm (IA in FIG. 5) may be used in the similar image identification process of the execution module (330) described later, and the generation algorithm (GA in FIG. 5) may be used in the reinforcement caption generation process of the execution module (330) described later. A detailed description of each algorithm will be provided later.

[0083] Here, weights can be set for the edges connecting the nodes of each layer. The presence or absence of such weights or edges can be added, removed, or updated during the learning process. Therefore, through the learning process, the weights of the nodes and edges placed between k input nodes and i output nodes can be updated.

[0084] Before a deep learning algorithm (DA) performs training, all nodes and edges can be set to initial values. However, as information is input cumulatively, the weights of the nodes and edges change, and in this process, a matching can be achieved between the parameters input as learning factors and the values ​​assigned to the output nodes.

[0085] Additionally, when using a cloud server, the deep learning algorithm (DA) can receive and process a large number of parameters. Therefore, the deep learning algorithm (DA) can perform learning based on vast amounts of data.

[0086] The weights of nodes and edges between input nodes and output nodes constituting a deep learning algorithm (DA) can be updated through the learning process of the deep learning algorithm. In addition, parameters that are input or output in the deep learning algorithm (DA) can be further modified and expanded with various data.

[0088] Referring again to FIGS. 1 and FIGS. 2, the database module (320) can transmit these comparison images (CI) and deep learning algorithms (DA) to other components within the image caption generation device (300). For example, the database module (320) can transmit the comparison images (CI) and deep learning algorithms (DA) to the execution module (330).

[0089] The execution module (330) can perform the image caption generation task.

[0090] For some examples, the execution module (330) can generate an image caption for the target image (TI) based on the target image (TI), the comparison image (CI), and the deep learning algorithm (DA).

[0091] For example, the execution module (330) can determine a similar image among the comparison images (CI) that has a high similarity to the target image (TI), and generate an image caption for the target image based on the image caption of the determined similar image.

[0092] Hereinafter, the image caption generation process of the execution module (330) will be explained with reference to FIGS. 4 to 7.

[0094] FIG. 4 is a flowchart of an image caption generation process according to some embodiment of the present invention. FIG. 5 is a conceptual diagram for explaining an image caption generation process according to some embodiment of the present invention. FIG. 6 is a detailed flowchart of the step of determining a similar image of FIG. 4. FIG. 7 is a detailed flowchart of the step of generating an enhancement caption of FIG. 4. Each step (S100 to S400) of FIG. 4, each step (S210 to S240) of FIG. 6, and each step (S310 and S320) of FIG. 7 can be performed by the execution module (330) of FIG. 2.

[0095] Referring to FIGS. 2, FIGS. 4 and FIGS. 5, first, the execution module (330) can receive a target image (TI) (S100).

[0096] In some examples, the execution module (330) may receive a target image (TI) from the communication module (310). However, embodiments of the present invention are not limited thereto, and the execution module (330) may also receive a target image (TI) from the database module (320).

[0097] Next, the execution module (330) can determine a similar image (hereinafter referred to as "SI") based on a target image (TI) and a target caption (hereinafter referred to as "TC") for the target image (TI) (S200).

[0098] The target caption (TC) is an image caption (caption1 to caption) for the target image (TI) up to a specific time point (T(t)) among multiple time points where the image caption generation task is performed. t It may include ). That is, the target caption (TC) may include at least one image caption (caption1 to caption t...may include .... In other words, the target caption (TC) at a specific point in time (T(t)) may include the image caption (caption1) at the first point in time (T(1)), and the image caption (not shown) at the second point in time (T(2)) generated through the process of steps (S100) to (S400) with respect to the image caption (caption1) at the first point in time (T(1)).

[0099] In some examples, the execution module (330) can determine a similar image (SI) based on a target image (TI), a target caption (TC), and an identification algorithm (hereinafter referred to as "IA").

[0100] At this time, the identification algorithm (IA) may include an algorithm based on a Vision-Language Model (VLM). A VLM is a model that processes text and images together. A VLM operates by understanding visual content and natural language together and learning the interaction between these two types of information. That is, a VLM operates based on the fact that images and text can share the same feature space. For example, it may include a Contrastive Language-Image Pre-training (CLIP) model, but embodiments of the present invention are not limited thereto, and the VLM may include a Vision-and-Language BERT (VILBERT) model, a VisualBERT model, etc. At this time, the identification algorithm (IA) may be stored in advance in a database module (320).

[0101] The identification algorithm (IA) may include a text embedding part of the identification algorithm (hereinafter referred to as "IA_T") that receives text data (target caption (TC)) as input and performs embedding, and an image embedding part of the identification algorithm (hereinafter referred to as "IA_I") that receives image data (target image (TI)) as input and performs embedding.

[0102] More specifically, referring to FIGS. 2, 4, 5 and 6, first, the execution module (330) can generate a text embedding (hereinafter referred to as "EM_T") by inputting a target caption (TC) into the text embedding part (IA_T) of the identification algorithm (IA) (S210).

[0103] Next, the execution module (330) can generate an image embedding (hereinafter referred to as "EM_I") by inputting a comparison image (CI) into the image embedding section (IA_I) of the identification algorithm (IA) (S220). At this time, the comparison image (CI) may be multiple, and the execution module (330) can generate multiple image embeddings (EM_I) by performing embedding on each of the multiple comparison images (CI).

[0104] Next, the execution module (330) can calculate the similarity between the text embedding (EM_T) and the image embedding (EM_I) (S230). As some examples, the execution module (330) can calculate the cosine similarity between the text embedding (EM_T) and the image embedding (EM_I). However, embodiments of the present invention are not limited thereto, and the execution module (330) may calculate Euclidean distance, Manhattan distance, Jaccard similarity, Pearson correlation coefficient, etc., other than cosine similarity.

[0105] Next, the execution module (330) can identify similar images (SI) based on the calculated similarity (S240). In some examples, the execution module (330) can determine the image with the maximum calculated similarity as the similar image (SI).

[0106] Referring again to FIGS. 2, FIGS. 4 and FIGS. 5, the execution module (330) can then generate a reinforcement caption (hereinafter referred to as “RC”) for a target image (TI) based on a similar caption (hereinafter referred to as “SC”) for a similar image (SI) (S300).

[0107] At this time, the similar caption (SC) may be an image caption for the similar image (SI). Although FIG. 5 is illustrated with five similar captions (SC) for the similar image (SI) for convenience of explanation, it is obvious that embodiments of the present invention are not limited thereto.

[0108] In some examples, the execution module (330) can generate an enhancement caption (RC) based on a target image (TI), a similar caption (SC), and a generation algorithm (hereinafter referred to as "GA").

[0109] At this time, the generation algorithm (GA) may include an algorithm based on the Large Vision-Language Model (LVLM). At this time, the LVLM may be a model that combines the Vision-Language Model (VLM) and the Large Language Model (LLM). In other words, the LVLM may be a VLM model that includes the LLM. That is, the LVLM may be a model that combines the VLM and the LLM. At this time, the generation algorithm (GA) in the form of an LMLM may be stored in advance in the database module (320).

[0110] VLM is a model that processes text and images together. VLM operates by understanding visual content and natural language together and learning the interaction between these two types of information. In other words, VLM operates based on the fact that images and text can share the same feature space. For example, it may include the CLIP (Contrastive Language-Image Pre-training) model, but embodiments of the present invention are not limited thereto, and VLM may include the VilBERT (Vision-and-Language BERT) model, the VisualBERT model, etc.

[0111] LLM is trained based on large-scale text data and is primarily used for language understanding and generation. LLM has a very large number of parameters and can apply generalized knowledge across various language processing tasks. LLM is utilized in various Natural Language Processing (NLP) tasks such as translation, summarization, text generation, sentence completion, and dialogue generation. As some examples, LLM may include the Chat-GPT model, but embodiments of the present invention are not limited thereto, and LLM may include the BERT (Bidirectional Encoder Representations from Transformers) model, the T5 (Text-to-Text Transfer Transformer) model, etc. In this case, the generation algorithm (GA) may be stored in advance in the database module (320).

[0112] More specifically, referring to FIGS. 2, 4, 5 and 7, first, the execution module (330) can determine a selected caption (Chosen Caption, hereinafter referred to as "CC") based on similar captions (SC) (S310). In some examples, the execution module (330) may randomly extract a predetermined number of similar captions from a plurality of similar captions (SC) and determine the extracted similar captions as the selected captions (CC). For example, the execution module (330) may determine two similar captions (SC) randomly extracted from a plurality of similar captions (SC) as the selected captions (CC), but it is obvious that embodiments of the present invention are not limited thereto.

[0113] Next, the execution module (330) can generate an augmented caption (RC) using a selected caption (CC), a target image (TI), and a generation algorithm (GA) (S320). In some examples, the execution module (330) can generate an augmented caption (RC) by inputting request data requesting a visual distinction between the target image (TI) and the selected caption (CC), and between the target image (TI) and the selected caption (CC), into the generation algorithm (GA). At this time, the generation algorithm (GA) can extract features from each of the target image (TI) and the selected caption (CC), generate an augmented caption (RC) that further details the features of the target image (TI) based on the differences between the extracted features, and transmit the generated augmented caption (RC) to the execution module (330).

[0114] Referring again to FIGS. 2, FIGS. 4 and FIGS. 5, the execution module (330) can then add an reinforcement caption (RC) to the target caption (TC) (S400).

[0115] In other words, the execution module (330) uses the reinforcement caption (RC) generated at the above specific time point (T(t)) as an image caption (caption t+1 It can be added to the target caption (TC) as ). In this case, the target caption (TC) consists of multiple image captions (caption1 to caption t+1 It will include ).

[0116] Next, the execution module (330) can repeat steps (S200 to S400) (S500).

[0117] For example, the execution module (330) reflects the reinforcement caption (RC) at a specific point in time (T(t)) so that multiple image captions (caption1 to caption) t+1 Steps (S200 to S400) can be performed again using a target caption (TC) that includes ).

[0118] Through this, the image caption generation device (300) according to some embodiments of the present invention can achieve a virtuous cycle structure in which, by repeating the process of searching for a similar image (SI) and the process of generating a caption (enhanced caption (RC)) through comparison with the similar image (SI), the more well the target caption (TC) used when searching for the similar image (SI) explains the target image (TI), the more similar images (SI) with higher similarity are searched, and the more differences from the target image (TI) are asked by searching for similar images (SI) with high similarity, the more the target caption (TC) is generated by focusing on more detailed and minor parts. That is, the image caption (caption) generated at a specific point in time (T(t)) t+1 The target caption (TC) will better explain the target image (TI) based on ), and during the image caption generation process at a subsequent time point (e.g., T(t+1)), a similar image (SI) that is more similar to the target image (TI) will be searched based on this target caption (TC), and an augmented caption (e.g., caption) generated based on this similar image (SI) t+2 Since ) will describe the features of the target image (TI) in more detail, the image caption generation device (300) according to some embodiments of the present invention can generate a more accurate image caption for the target image (TI) through the above iteration process (S500).

[0120] FIG. 8 is a diagram illustrating a hardware implementation of an image caption generation device according to some embodiments of the present invention.

[0121] Referring to FIGS. 1 and FIGS. 8, an image caption generation device (300) according to some embodiments of the present invention may be implemented as an electronic device (1000). The electronic device (1000) may include a controller (1010), an input / output device (1020, I / O), a memory device (1030, memory device), an interface (1040, interface), and a bus (1050, bus). The controller (1010), the input / output device (1020), the memory device (1030), and / or the interface (1040) may be connected to each other through the bus (1050). In this case, the bus (1050) corresponds to a path through which data is moved.

[0122] Specifically, the controller (1010) may include at least one of a CPU (Central Processing Unit), MPU (Micro Processor Unit), MCU (Micro Controller Unit), GPU (Graphic Processing Unit), microprocessor, digital signal processor, microcontroller, application processor (AP), and logic elements capable of performing similar functions.

[0123] The input / output device (1020) may include at least one of a keypad, a keyboard, a touchscreen, and a display device.

[0124] The memory device (1030) can store data and / or programs, etc.

[0125] The interface (1040) can perform the function of transmitting data to a communication network or receiving data from a communication network. The interface (1040) may be wired or wireless. For example, the interface (1040) may include an antenna or a wired / wireless transceiver, etc. Although not illustrated, the memory device (1030) may further include high-speed DRAM and / or SRAM, etc. as an operational memory to enhance the operation of the controller (1010). The memory device (1030) may store a program or application internally.

[0126] An image caption generation device (300) according to embodiments of the present invention may be a system formed by connecting a plurality of electronic devices (1000) to each other through a network. In such cases, each module or combination of modules may be implemented as an electronic device (1000). However, the present embodiment is not limited thereto.

[0127] Additionally, the image caption generating device (300) may be implemented as at least one of a workstation, a data center, an internet data center (IDC), a direct attached storage (DAS) system, a storage area network (SAN) system, a network attached storage (NAS) system, a redundant array of inexpensive disks, or a redundant array of independent disks) system, and an Electronic Document Management (EDMS) system, but the present embodiment is not limited thereto.

[0128] Additionally, the image caption generation device (300) can transmit data to an external database (100) and / or a user terminal (200) via a network. The network may include a network based on wired internet technology, wireless internet technology, and local area communication technology. Wired internet technology may include, for example, at least one of a local area network (LAN) and a wide area network (WAN).

[0129] Wireless internet technology may include, for example, at least one of Wireless LAN (WLAN), DMNA (Digital Living Network Alliance), WiBro (Wireless Broadband), WiMAX (World Interoperability for Microwave Access), HSDPA (High Speed ​​Downlink Packet Access), HSUPA (High Speed ​​Uplink Packet Access), IEEE 802.16, Long Term Evolution (LTE), LTE-A (Long Term Evolution-Advanced), Wireless Mobile Broadband Service (WMBS), and 5G NR (New Radio) technology. However, the present embodiment is not limited thereto.

[0130] Short-range communication technology may include, for example, at least one of Bluetooth, RFID (Radio Frequency Identification), Infrared Data Association (IrDA), Ultra-Wideband (UWB), ZigBee, Near Field Communication (NFC), Ultra Sound Communication (USC), Visible Light Communication (VLC), Wi-Fi, Wi-Fi Direct, and 5G NR (New Radio). However, the present embodiment is not limited thereto.

[0131] An image caption generation device (300) communicating through a network may comply with technical standards and standard communication methods for mobile communication. For example, a standard communication method may include at least one of GSM (Global System for Mobile communication), CDMA (Code Division Multi Access), CDMA2000 (Code Division Multi Access 2000), EV-DO (Enhanced Voice-Data Optimized or Enhanced Voice-Data Only), WCDMA (Wideband CDMA), HSDPA (High Speed ​​Downlink Packet Access), HSUPA (High Speed ​​Uplink Packet Access), LTE (Long Term Evolution), LTEA (Long Term Evolution-Advanced), and 5G NR (New Radio). However, the present embodiment is not limited thereto.

[0132] The above description is merely an illustrative explanation of the technical concept of the present embodiment, and a person skilled in the art to which the present embodiment belongs would be able to make various modifications and variations within the scope of the essential characteristics of the present embodiment. Accordingly, the present embodiments are intended to explain, not limit, the technical concept of the present embodiment, and the scope of the technical concept of the present embodiment is not limited by these embodiments. The scope of protection of the present embodiment shall be interpreted by the claims below, and all technical concepts within an equivalent scope shall be interpreted as being included within the scope of rights of the present embodiment.

Claims

Claim 1 A method for generating an image caption performed by an image caption generation device comprises: (a) receiving a target image; (b) determining a similar image for the target image based on the target image and a target caption which is an image caption for the target image; (c) generating a reinforcement caption for the target image based on the similar caption which is an image caption for the similar image; and (d) adding the reinforcement caption to the target caption. The image caption generation method comprises a step (e) of repeating steps (b) through (d), wherein step (c) includes determining a predetermined number of selected captions among the similar captions and inputting request data requesting visual distinction between the target image, the selected captions, and the target image and the selected captions into a generation algorithm based on a Large Vision-Language Model (LVLM) to generate the reinforcement captions, and step (e) determines the similar image based on the target caption including the reinforcement captions added at the first time point when step (b) is performed at a second time point after the first time point, after the reinforcement captions generated by performing steps (b) through (d) at a first time point are added to the target captions. Claim 2 An image caption generation method according to claim 1, wherein step (b) includes the step of identifying similar images for a target image based on a predefined identification algorithm. Claim 3 In claim 2, the identification algorithm includes an algorithm based on a Vision-Language Model (VLM), and the VLM includes a Contrastive Language-Image Pre-training (CLIP) model, in an image caption generation method. Claim 4 In claim 2, the identification algorithm comprises a text embedding unit and an image embedding unit, and the step of identifying a similar image for a target image based on the identification algorithm comprises: a step of generating a text embedding by inputting the target caption into the text embedding unit, and a step of identifying the similar image based on the similarity between the text embedding and a plurality of comparison images stored in advance. Claim 5 In claim 4, the step of identifying the similar image based on the similarity between the text embedding and a plurality of pre-stored comparison images comprises: a step of generating a plurality of image embeddings by inputting each of the plurality of comparison images into the image embedding unit; and a step of identifying the similar image based on the cosine similarity between the text embedding and the image embedding. Claim 6 delete Claim 7 An image caption generation method according to claim 1, wherein determining a predetermined number of selected captions among the similar captions includes randomly extracting two similar captions from a plurality of similar captions and determining them as the selected captions. Claim 8 delete Claim 9 delete Claim 10 delete

Citation Information

Patent Citations

  • Intelligent image captioning

    US20170098153A1

  • Image captioning

    US20230153522A1