Tagging method for content and system therefor

Abstract

Provided are a tagging method for content and a system therefor. A tagging method according to some embodiments may comprise the steps of: generating a candidate tag set including a plurality of candidate tags related to content through an image analysis module; configuring a prompt on the basis of the candidate tag set, context data for the content, and a query statement; and generating a tag set for the content on the basis of a response of a language model to the prompt. According to the method, a high-quality tag that well represents the characteristics, details, and the like of the content can be easily created, and a tagging operation for the content can easily be automated.

Description

Tagging method and system for content

[0001] The present disclosure relates to a method and system for automatically performing tagging of content.

[0002] OTT (Over The Top) service providers perform tagging, which involves assigning keywords that describe the characteristics and content of the content as tags. These tags are then used for various purposes, including content search, grouping similar content, and providing information.

[0003] However, most OTT service providers still perform this tagging process manually. Furthermore, this manual tagging method incurs significant human costs, given the continuous production of new content in OTT services.

[0004] A technical problem to be solved through some embodiments of the present disclosure is to provide a method and system capable of accurately performing automatic tagging of content.

[0005] The technical problems of the present disclosure are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art of the present disclosure from the description below.

[0006] According to some embodiments of the present disclosure for solving the above-described technical problem, a tagging method for content, which is performed by at least one computing device, may include: generating a candidate tag set including a plurality of candidate tags related to content through an image analysis module; configuring a prompt based on the candidate tag set, context data for the content, and a query statement; and generating a tag set for the content based on a response of a language model to the prompt.

[0007] In some embodiments, the context data includes metadata of the content, wherein the metadata may include at least one of a title, a genre, and a summary of the content.

[0008] In some embodiments, the context data may include a script of the content.

[0009] In some embodiments, the image analysis module includes an object detection model and an action recognition model, and the step of generating the candidate tag set may include a step of detecting an object appearing in the content through the object detection model and generating a first candidate tag representing the detected object, and a step of recognizing an action of an object appearing in the content through the action recognition model and generating a second candidate tag representing the recognized action.

[0010] In some embodiments, the query may include a phrase requesting that significant tags matching the content be selected from the candidate tag set by referencing the context data.

[0011] In some embodiments, the step of configuring the prompt may include the steps of: obtaining specific content tagged with a set of correct tags; extracting candidate tags from the specific content; configuring a first prompt having the extracted candidate tags, context data of the specific content, and a query as prompt elements; evaluating a first response of the language model to the first prompt using the set of correct tags; configuring a second prompt by changing at least one of an arrangement and a content of the prompt elements; evaluating a second response of the language model to the second prompt using the set of correct tags; generating a prompt template for the language model using a result of evaluating the first response and a result of evaluating the second response; and configuring the prompt using the prompt template.

[0012] In some embodiments, the response of the language model includes at least one tag, and the step of generating the tag set may include the step of removing a tag that is not present in the candidate tag set from among the at least one tag, and constructing the tag set using the remaining tags.

[0013] In some embodiments, the tagging method may further include the steps of constructing a tag discriminator using the candidate tag set and the tag set, the tag discriminator being a model that determines whether an input candidate tag is an important tag that matches the relevant content, obtaining additional content, and performing tagging on the additional content using the image analysis module to which the tag discriminator is added without the aid of the language model.

[0014] In some embodiments, the tag determiner may be configured to further receive image features of the relevant content extracted through the image analysis module and determine whether the input candidate tag is an important tag.

[0015] In some embodiments, the tag set is tagged to the content, and the tagging method may further include a step of creating a content collection for the common tag by grouping other contents having a common tag with the content among a plurality of previously registered contents.

[0016] According to some embodiments of the present disclosure for solving the above-described technical problem, a tagging system includes a memory storing one or more processors and a computer program executed by the one or more processors, wherein the computer program may include instructions for an operation of generating a candidate tag set including a plurality of candidate tags related to content through an image analysis module, an operation of configuring a prompt based on the candidate tag set, context data for the content, and a query statement, and an operation of generating a tag set for the content based on a response of a language model to the prompt.

[0017] According to some embodiments of the present disclosure for solving the above-described technical problem, a computer program may be stored in a computer-readable recording medium to be combined with a computing device and execute the steps of: generating a candidate tag set including a plurality of candidate tags related to content through an image analysis module; configuring a prompt based on the candidate tag set, context data for the content, and a query sentence; and generating a tag set for the content based on a response of a language model to the prompt.

[0018] According to some embodiments of the present disclosure, a set of candidate tags for content is generated through an image analysis module, and important tags (i.e., tags that match the content) are selected from the set of candidate tags through a language model that understands the context (or content) of the content. In this case, high-quality tags that well represent the characteristics and content of the content can be easily generated (extracted), and the tagging process can be easily automated. Naturally, this can significantly reduce the human and time costs associated with the tagging process.

[0019] In addition, since the image analysis module is configured to include an object detection model, an action recognition model, etc., various candidate tags for the content can be easily and without omission extracted.

[0020] Furthermore, by constructing a language model prompt based on a query requesting the selection of important tags within a set of candidate tags, the hallucination problem of the language model can be easily prevented. For example, the problem of generating tags that do not accurately match the content due to hallucination in the language model can be easily prevented.

[0021] Additionally, the hallucination problem of the language model can be more easily prevented by verifying the tags included in the language model's response using a set of candidate tags.

[0022] Additionally, prompt templates optimized for language models can be automatically generated using content that has a set of correct tags, resulting in further improved tag quality.

[0023] Furthermore, the image analysis module can be enhanced by using the tag set of content obtained through the language model as training data. For example, a tag discriminator can be built using the training data to determine whether candidate tags are important tags that match the content, and this tag discriminator can be added to the image analysis module. Furthermore, the enhanced image analysis module can perform content tagging without the assistance of a language model. In this case, the resources required for tagging can be significantly reduced, and the hallucination problem of the language model can be completely eliminated.

[0024] The effects according to the technical idea of ​​the present disclosure are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

[0025] FIG. 1 is an exemplary diagram schematically illustrating the operation of a tagging system according to some embodiments of the present disclosure.

[0026] FIG. 2 is an exemplary diagram to further explain the operation of a tagging system according to some embodiments of the present disclosure.

[0027] FIG. 3 is an exemplary flowchart illustrating a tagging method for content according to some embodiments of the present disclosure.

[0028] FIG. 4 is an exemplary diagram illustrating a method for extracting candidate tags according to some embodiments of the present disclosure.

[0029] FIGS. 5 and 6 illustrate prompt templates that may be referenced in some embodiments of the present disclosure.

[0030] FIG. 7 is an exemplary diagram illustrating a method for generating a prompt template according to some embodiments of the present disclosure.

[0031] FIG. 8 is an exemplary diagram illustrating a tag verification method according to some embodiments of the present disclosure.

[0032] FIG. 9 and FIG. 10 are exemplary diagrams illustrating a tagging module / model operation method according to some embodiments of the present disclosure.

[0033] FIG. 11 is an exemplary diagram illustrating a tagging module / model operation method according to some other embodiments of the present disclosure.

[0034] FIG. 12 and FIG. 13 are exemplary drawings for explaining a method of creating a content collection according to some application examples of the present disclosure.

[0035] FIG. 14 illustrates an exemplary computing device that may implement a tagging system according to some embodiments of the present disclosure.

[0036] Hereinafter, various embodiments of the present disclosure will be described in detail with reference to the attached drawings. The advantages and features of the present disclosure, and methods for achieving them, will become clear with reference to the embodiments described in detail below together with the attached drawings. However, the technical idea of ​​the present disclosure is not limited to the following embodiments and may be implemented in various different forms. The following embodiments are provided only to complete the technical idea of ​​the present disclosure and to fully inform those skilled in the art of the present disclosure of the scope of the present disclosure, and the technical idea of ​​the present disclosure is defined only by the scope of the claims.

[0037] In describing various embodiments of the present disclosure, if it is determined that a detailed description of a related known configuration or function may obscure the gist of the present disclosure, the detailed description will be omitted.

[0038] Unless otherwise defined, the terms (including technical and scientific terms) used in the following examples may be used with meanings commonly understood by those of ordinary skill in the art to which this disclosure pertains; however, this may vary depending on the intentions of engineers working in the relevant field, precedents, the emergence of new technologies, etc. The terminology used in this disclosure is for the purpose of describing the embodiments and is not intended to limit the scope of this disclosure.

[0039] In the following examples, singular expressions include plural concepts unless the context clearly specifies that they are singular. Furthermore, plural expressions include singular concepts unless the context clearly specifies that they are plural.

[0040] In addition, terms such as first, second, A, B, (a), (b), etc. used in the following embodiments are only used to distinguish certain components from other components, and the nature, order, or sequence of the components are not limited by the terms.

[0041] Hereinafter, various embodiments of the present disclosure will be described in detail with reference to the attached drawings.

[0042] FIG. 1 is an exemplary drawing schematically illustrating the operation of a tagging system (10) according to some embodiments of the present disclosure.

[0043] As illustrated in FIG. 1, the tagging system (10) is a computing device / system capable of automatically generating tags for content (11). For example, the tagging system (10) can automatically generate a tag set (13) that matches the content (11) by using context data (12) of the content (11). The tag set (13) thus generated can be tagged (set) to the content (11). The tag set (13) can include at least one tag (e.g., keyword tag, etc.), and FIG. 1 illustrates an example in which the tag set (13) is composed of multiple tags.

[0044] A tag may be, for example, a keyword indicating the content (11), characteristics (e.g., topic, atmosphere, director, character, actor, production date, etc.). However, the scope of the present disclosure is not limited thereto. For example, a tag may be a keyword indicating other information, or may take the form of a phrase, sentence, etc. In some cases, a tag may also be named with terms such as "label" or "annotation."

[0045] Content (11) is content with visual elements (or content capable of image analysis), and may include various types / forms of content without limitation. For example, content (11) may be video content, image content, etc., but the scope of the present disclosure is not limited thereto.

[0046] Context data (12) refers to data (e.g., text data) that describes the content (11) or helps in understanding (e.g., understanding the context / content) of the content (11). The context data (12) may include, for example, metadata and scripts of the content (11). However, the scope of the present disclosure is not limited thereto. The context data (12) may also be named as "description data," "reference data," or "description data," depending on the case.

[0047] The metadata of the content (11) may include various information related to the content (11) without limitation, such as the title, genre, running time, director, actors, characters, production date, summary, reviews, etc. of the content (11). In some cases, the script of the content (11) may also be included in the category of metadata.

[0048] The script of content (11) refers to text that describes the details of the content (11). The script of content (11) may be, for example, a subtitle, a caption, a synopsis, a scenario, etc., but the scope of the present disclosure is not limited thereto. The script may be prepared in advance or may be generated through a speech-to-text or speech recognition technique.

[0049] The detailed operation of the tagging system (10) is illustrated in FIG. 2. As illustrated in FIG. 2, the tagging system (10) can generate a tag set (13) of content (11) using an image analysis module (21) and a language model (22). More specifically, the tagging system (10) can generate a candidate tag set (23) including a plurality of candidate tags (illustrated as 'tags' in the drawing) through the image analysis module (21), and can analyze the candidate tag set (23) and context data (12) through the language model (22) to generate (determine) a final tag set (13) of content (11). Since the language model (22) can sufficiently understand the context data (12), important tags (e.g., tags that well represent the characteristics and content of the content) that match the content (11) can be accurately selected from the candidate tag set (23) by using the language model (22). In some cases, the tagging system (10) may control the language model (22) to generate tags that match the content (11) based on a set of candidate tags (23) and / or context data (12) (e.g., by entering a prompt requesting tag generation).

[0050] For reference, a language model (22) refers to a neural network model that has acquired universal understanding of language (or natural language / text) by learning a large amount of texts (e.g., texts from various domains). The language model (22) can be viewed as a large-scale model with question and answer capabilities based on a text interface, or as a model that can 'generate' responses to questions. Therefore, it may be named as a 'largescale language model (LLM)', 'generative model', 'question-answering model', 'conversational model', 'tagging model', etc. depending on the case. In addition, the term 'model' may be used interchangeably with terms such as 'module', 'AI (Artificial Intelligence)', etc.

[0051] The specific method by which the tagging system (10) performs automatic tagging will be described in more detail with reference to the drawings below in FIG. 3.

[0052] In some cases, the tagging system (10) may provide a set of tags (13) for content (11) to a separate device (not shown).

[0053] Additionally, in some cases, the tagging system (10) may perform tagging tasks only using the enhanced image analysis module (21) after gradually strengthening the image analysis module (21) using a tag set (13) generated with the help of a language model (22). Further reference is made to the descriptions of FIGS. 9 to 11 for this purpose.

[0054] Additionally, in some cases, the tagging system (10) may perform tagging operations on various contents and group contents with common tags to create a content collection. For further details, please refer to the descriptions of FIGS. 12 and 13.

[0055] The tagging system (10) described above may be implemented with at least one computing device. For example, all functions of the tagging system (10) may be implemented on a single computing device, or a first function of the tagging system (10) may be implemented on a first computing device and a second function may be implemented on a second computing device. Alternatively, specific functions of the tagging system (10) may be implemented on multiple computing devices.

[0056] A computing device may include any device having computing (processing) capabilities, and for an example of such a device, see FIG. 14.

[0057] Note that a computing device is a collection of interacting components (e.g., memory, processor, etc.), and thus may sometimes be referred to as a "computing system." Of course, the term "computing system" can also encompass the concept of a collection of interacting computing devices.

[0058] The operation of the tagging system (10) according to some embodiments of the present disclosure has been briefly described with reference to FIGS. 1 and 2. Hereinafter, various methods that can be performed in the tagging system (10) described above will be described with reference to the drawings including FIG. 3 and below.

[0059] For ease of understanding, the following description assumes that all steps / operations of the methods described below are performed in the aforementioned tagging system (10). Therefore, if the subject of a specific step / operation is omitted, it can be understood that it is performed in the tagging system (10). However, in an actual environment, some steps / operations of the methods described below may be performed in other computing devices.

[0060] Figure 3 is an exemplary flowchart schematically illustrating a tagging method for content according to some embodiments of the present disclosure. However, this is merely a preferred embodiment for achieving the objectives of the present disclosure, and it is understood that some steps may be added or deleted as needed.

[0061] As illustrated in FIG. 3, the tagging method according to the embodiments may begin with step S31, which involves acquiring content and its context data. The content may be, for example, video content comprising multiple frame images, but the scope of the present disclosure is not limited thereto. However, for ease of understanding, the following description will assume that the content is "video content." As described above, the context data may include, for example, content metadata and scripts.

[0062] In step S32, a candidate tag set for the corresponding content may be generated through the image analysis module (21). The candidate tag set may include multiple candidate tags related to the corresponding content.

[0063] The image analysis module (21) may be configured to include various types of modules / models that analyze images to extract information. For example, the image analysis module (21) may include a module based on an image analysis algorithm, an object detection model, an image captioning model (or scene recognition model), a scene segmentation model, an expression (emotion) recognition model, etc., but the scope of the present disclosure is not limited thereto. The image analysis module (21) may also be named, for example, a 'tagging module', depending on the case.

[0064] The specific method of generating a candidate tag set or extracting candidate tags in step S32 may vary depending on the embodiment.

[0065] In some embodiments, the tagging system (10) may analyze each frame image of content (i.e., video content) via an image analysis module (21) to extract candidate tags. For example, the tagging system (10) may extract candidate tags for each frame image. The candidate tags thus extracted may constitute a candidate tag set.

[0066] In some other embodiments, the tagging system (10) may select key frame images from among frame images of content (i.e., video content) and analyze the key frame images through an image analysis module (21) to extract candidate tags. For example, the tagging system (10) may select key frame images based on the number of objects, the degree of movement of the objects, the results of object action recognition, the results of scene recognition, the similarity between the results of scene recognition and the title of the content, etc. However, the scope of the present disclosure is not limited thereto.

[0067] In some other embodiments, the tagging system (10) can segment frame images of content (i.e., video content) by scene through a scene segmentation model. Furthermore, the tagging system (10) can extract candidate tags for each scene. For example, the tagging system (10) can select a representative frame image for each scene and analyze the representative frame image to extract candidate tags for the corresponding scene. Alternatively, the tagging system (10) can extract candidate tags from each frame image belonging to a scene and combine them to use them as candidate tags for the corresponding scene.

[0068] In some other embodiments, the tagging system (10) can detect objects appearing in content through an object detection model and determine (generate) tags (e.g., keywords) representing the detected objects as candidate tags for the corresponding content.

[0069] In some other embodiments, the tagging system (10) can recognize the behavior of an object appearing in content through a behavior recognition model and determine (generate) a tag (e.g., keyword) representing the recognized behavior as a candidate tag for the content.

[0070] In some other embodiments, the tagging system (10) may generate captions for content (e.g., for each frame image) through an image captioning model (or scene recognition model) and generate candidate tags for the content based on the captions. The candidate tags may be, but are not limited to, the caption itself, keywords extracted from the caption, or the results of sentiment analysis on the caption. For reference, the image captioning model refers to a model that outputs a caption (i.e., a description) of an input image as text in a natural language format.

[0071] In some other embodiments, a candidate tag set may be generated based on various combinations of the above-described embodiments. For example, as illustrated in FIG. 4 , the tagging system (10) may generate candidate tags associated with objects appearing in frame images of video content (43) through an object detection model (41) and may generate candidate tags associated with the actions of objects appearing in the frame images through an action recognition model (42). As a result, a candidate tag set (44) of the video content (43) may be generated.

[0072] This is explained again with reference to Figure 3.

[0073] In step S33, a prompt of the language model (22) may be constructed based on a candidate tag set, context data, and a query statement. For example, the tagging system (10) may construct a prompt by arranging prompt elements, such as a candidate tag set, context data, and a query statement, according to a predefined prompt template. The prompt template will be further described later with reference to FIGS. 5 to 7.

[0074] The specific way in which the prompt is configured in step S33 may vary depending on the embodiment.

[0075] In some embodiments, the tagging system (10) may divide the candidate tag set into a plurality of subsets based on a determination that the size of the candidate tag set (e.g., total number of candidate tags, total length, etc.) is greater than or equal to a threshold. Then, the tagging system (10) may configure a plurality of prompts based on each of the plurality of subsets. For example, the tagging system (10) may configure a first prompt based on a first subset, a common query, and first context data, and may configure a second prompt based on a second subset, a common query, and second context data. At this time, the first context data may be the same as the second context data, or at least a portion thereof may be different from each other. For example, the first context data may include a script of a portion of the entire script related to the first subset (e.g., a script of a frame image (or scene) from which tags of the first subset are extracted, a script of an adjacent frame image (or scene), etc.), and the second context data may include a script of a portion of the entire script related to the second subset.

[0076] In some other embodiments, the language model (22) may be a model with a maximum prompt length (e.g., maximum number of tokens). In such cases, the tagging system (10) may divide the candidate tag set into multiple subsets based on the maximum prompt length, and construct multiple prompts (i.e., prompts shorter than the maximum length) based on each of the subsets.

[0077] In some other embodiments, the prompt may be constructed based on various combinations of the embodiments described above.

[0078] Below, the prompt template will be described with reference to FIGS. 5 to 7.

[0079] FIGS. 5 and 6 illustrate prompt templates (50, 60) that may be referenced in some embodiments of the present disclosure.

[0080] As illustrated in FIG. 5, a prompt template (50) refers to a template that defines the prompt elements (51 to 54) to be included in the prompt, their arrangement order, the content of the query statement (53), etc. FIG. 5 illustrates an example in which a content title (51), a set of candidate tags (52), a query statement (53), and context data (54) of the content are defined as prompt elements (refer to the order of the prompt elements illustrated in FIG. 5 for the arrangement order).

[0081] The query (53) can be written to include, for example, a phrase requesting selection of important tags matching the content from the candidate tag set (52) by referencing the context (54) of the content. When the query (53) is written in this way, the hallucination problem of the language model (22) can be easily prevented. In other words, by explicitly requesting the language model (22) to select important tags from the candidate tag set (52), the problem of the language model (22) generating and outputting tags (e.g., keywords) that do not exist in the candidate tag set (52) can be easily prevented.

[0082] For an example of a prompt template, refer to the prompt template (60) illustrated in Fig. 6.

[0083] Meanwhile, the process of defining and designing a prompt template optimized for a language model (22) can be performed manually or in an automated manner.

[0084] For example, prompt templates can be predefined by a prompt engineer. For example, the prompt engineer can write a query requesting the selection of a specified number of important tags (e.g., the three most important tags) from a set of candidate tags so that the language model's (22) responses can be quickly and accurately evaluated. The prompt engineer can then input various prompts based on the query into the language model (22) and evaluate the language model's (22) responses to each prompt. The prompt engineer can then synthesize the evaluation results to define a prompt template optimized for the language model (22).

[0085] As another example, a prompt template can be automatically generated using at least one content that is tagged (set) with a set of correct tags. Specifically, as illustrated in FIG. 7, the tagging system (10) can extract candidate tags (73, i.e., a set of candidate tags) from content (71) given a set of correct tags (72). Next, the tagging system (10) can input various forms of prompts (e.g., 74, 76) based on the candidate tags (73) into the language model (22), and can evaluate the response (e.g., see 75, 77) of the language model (22) using the set of correct tags (72) (e.g., evaluating how well it matches the set of correct tags (72)). For example, the tagging system (10) can construct a first prompt (e.g., 74) based on candidate tags (73), context data of content (71), and a query sentence, and evaluate a response (e.g., 75) of a language model (22) to the first prompt (e.g., 74) using a set of correct tags (72). In addition, the tagging system (10) can construct a second prompt (e.g., 76) by changing at least one of the arrangement and content (e.g., content of a query sentence, etc.) of prompt elements, and evaluate a response (e.g., 77) of a language model (22) to the second prompt (e.g., 76) using a set of correct tags (72). The tagging system (10) can repeat these processes for other contents, and can generate a prompt template optimized for the language model (22) by comprehensively considering the evaluation results (e.g., generating a prompt template using a prompt with the highest comprehensive evaluation score).

[0086] This is explained again with reference to Figure 3.

[0087] In step S34, a tag set for the corresponding content may be generated based on the response of the language model (22) to the prompt. For example, the tagging system (10) may determine the tags included in the response of the language model (22) as the tag set for the corresponding content, or may generate the tag set for the corresponding content through additional processing of the tags included in the response. Examples of such processing may include merging similar tags, expanding tags based on synonyms, and removing tags whose similarity with the content title (or other metadata) is below a threshold, but the scope of the present disclosure is not limited thereto.

[0088] Meanwhile, in some embodiments, as illustrated in FIG. 8, the tagging system (10) may verify tags (84) included in a response (i.e., a response to a prompt (83)) using a candidate tag set (82) of the content (81). In addition, the tagging system (10) may also generate (configure) a tag set of the content (81) based on the verified tags (84). This verification process may be understood as being intended to prevent a problem in which tag quality is deteriorated due to the hallucination phenomenon of the language model (22). For example, the tagging system (10) may remove tags that do not exist in the candidate tag set (82) among the tags (84) and generate a tag set of the content (81) using the remaining tags. Alternatively, the tagging system (10) may remove tags whose similarity to the candidate tag (see 82) is below a threshold value among the tags (84) and generate a tag set of the content (81) using the remaining tags.

[0089] Hereinafter, a tagging method for content according to some embodiments of the present disclosure has been described with reference to FIGS. 3 to 8 . As described above, a candidate tag set for content is generated through an image analysis module (21), and important tags (i.e., tags that match the content) can be selected from the candidate tag set through a language model (22) that understands the context (or content) of the content. In this case, high-quality tags that well represent the characteristics and content of the content can be easily generated (extracted), and the tagging task can be easily automated. Of course, this can significantly reduce the human and time costs required for the tagging task.

[0090] In addition, since the image analysis module (21) is configured to include an object detection model (41), an action recognition model (42), etc., various candidate tags for the content can be easily and without omission extracted.

[0091] Furthermore, by constructing a prompt for the language model based on a query requesting selection of important tags within a set of candidate tags, the hallucination problem of the language model (22) can be easily prevented. For example, the problem of tags not accurately matching the content being generated due to the hallucination phenomenon of the language model (22) can be easily prevented.

[0092] In addition, by verifying the tags included in the response of the language model (22) using the candidate tag set, the hallucination problem of the language model (22) can be more easily prevented.

[0093] Meanwhile, according to various embodiments of the present disclosure, the tagging system (10) can gradually strengthen the image analysis module (21) using a tag set generated with the help of a language model (22). Furthermore, when the image analysis module (21) is sufficiently strengthened, the tagging system (10) can perform tagging tasks on content using only the image analysis module (21) without the help of a language model (22). These embodiments will now be described with reference to FIGS. 9 to 11 .

[0094] FIG. 9 and FIG. 10 are exemplary drawings for explaining a tagging module / model operation method according to some embodiments of the present disclosure.

[0095] As illustrated in FIG. 9, from time point 'T1' to time point 'T2' (i.e., until the image analysis module (21) is sufficiently strengthened), the tagging system (10) can perform tagging work on content by using the image analysis module (21) and the language model (22) together. In addition, the tagging system (10) can strengthen the image analysis module (21) by using the tag set generated through the language model (22).

[0096] For example, as illustrated in FIG. 10, let us assume that the image analysis module (21) includes an object detection model (41), and a tag set (104, hereinafter referred to as “correct tag set”) for content (102) is generated through a language model (22). In this case, the tagging system (10) can train a tag discriminator (101) using the candidate tag set (103) and the correct tag set (104) generated through the object detection model (41) as training data. The tag discriminator (101) refers to a model (e.g., a neural network model such as a fully connected layer) that determines whether an input tag (e.g., a one-hot vector of a tag, an embedding of a tag, etc. may be input) is an important tag that matches the content (102). As illustrated, the tag discriminator (101) may be configured to further input image features of the content (102) (e.g., feature maps extracted through an object detection model (41) or an image analysis module (21)) in order to accurately determine whether the input tag matches the content (102). In addition, in some cases, the tag discriminator (101) may be configured to further input at least a portion of context data of the content (102). The tagging system (10) may determine (predict) whether a candidate tag input through the tag discriminator (101) is an important tag matching the content (102) and compare the determination result with a correct tag set (104) (e.g., determine whether the candidate tag exists in the correct tag set (104)) to calculate a loss (105, 'L'). In addition, the tagging system (10) can update the weight parameters of the tag discriminator (101) based on the loss (105). By repeatedly performing these processes for other candidate tags and other contents within the candidate tag set (103), a tag discriminator (101) equipped with the ability to discriminate important tags can be constructed.And, by adding the tag discriminator (101) constructed in this way to the image analysis module (21), the image analysis module (21) can be strengthened.

[0097] This is explained again with reference to Figure 9.

[0098] When the 'T2' point is reached, the tagging system (10) can perform tagging work on content using only the enhanced image analysis module (21, e.g., the image analysis module (21) with the tag discriminator (101) added). In this case, the resources invested in the tagging work can be significantly reduced (because the language model (22) requiring considerable resources is not used), and the hallucination problem of the language model (22) can be completely freed from.

[0099] The 'T2' point in time may be, for example, a point in time when the performance of the enhanced image analysis module (21) becomes higher than a reference value, and the performance of the enhanced image analysis module (21) may be judged (measured) using a tag set generated through a language model (22). For example, the tagging system (10) may judge the performance of the image analysis module (21) based on the difference between the first tag set generated through the language model (22) and the second tag set generated through the image analysis module (21) (e.g., a smaller difference means that the performance of the enhanced image analysis module (21) is excellent enough to approach the language model (22)).

[0100] In some cases, the tagging system (10) may verify the tag set of the content obtained through the enhanced image analysis module (21) using a language model (22). For example, the tagging system (10) may request the language model (22) to determine (verify) whether the tag set matches the context data of the content (e.g., construct a prompt based on a caption, context data, and a query requesting verification), or may perform verification by directly comparing the tag set with a tag set generated through the language model (22).

[0101] FIG. 11 is an exemplary diagram illustrating a tagging module / model operation method according to some other embodiments of the present disclosure.

[0102] As illustrated in FIG. 11, from time point 'T1' to time point 'T2', the tagging system (10) can perform tagging tasks on content by using an object detection model (41) and a language model (22) together. Of course, the tagging system (10) can also perform tagging tasks by using other models (e.g., an action recognition model (42)).

[0103] More specifically, the tagging system (10) can generate a tag set for each frame image or scene through a language model (22), and can convert the tag set into a caption in a natural language form by using the language model (22) again. In addition, the tagging system (10) can build an image captioning model (111) using the frame image of the content (or frame images belonging to the scene) and captions as training data. The image captioning model (111) built in this way can be added to the image analysis module (21), and as a result, the image analysis module (21) can be strengthened.

[0104] When the 'T2' time point is reached, the tagging system (10) can perform tagging tasks by using the image captioning model (111) and the language model (22) together. For example, the tagging system (10) can configure a prompt based on the caption generated through the image captioning model (111), the context data of the content, and a query (e.g., a query requesting selection of important tags by referring to the context of the content or requesting processing of the caption), and can generate a tag set of the content based on the response of the language model (22) to the prompt. In addition, the tagging system (10) can perform additional learning on the image captioning model (111) in a similar manner as described above. As a result, the image captioning model (111) can be strengthened.

[0105] The 'T2' point in time may be, for example, a point in time when the performance of the image captioning model (111) exceeds a reference value, and the performance of the image captioning model (111) may be judged using the caption generated by the language model (22) (see the description in FIG. 9). However, the scope of the present disclosure is not limited thereto.

[0106] When the 'T3' point is reached, the tagging system (10) can perform tagging work on the content using only the enhanced image captioning model (111). For example, the tagging system (10) can generate a caption for the content (e.g., a caption for a frame image) through the enhanced image captioning model (111) and generate a set of tags for the content based on the caption (e.g., extracting keywords from the caption, etc.). The enhanced image captioning model (111) can accurately generate a caption that well represents the content, etc. of the content without the help of a language model (22) (or context data of the content) through an accurate understanding of the frame image.

[0107] The 'T3' time point can also be determined based on the performance of the enhanced image captioning model (11), similar to the 'T2' time point, for example. However, the scope of the present disclosure is not limited thereto.

[0108] In some cases, the tagging system (10) may verify the caption (or tag set) of the content obtained through the enhanced image captioning model (111) using the language model (22). For example, the tagging system (10) may request the language model (22) to determine (verify) whether the caption (or tag set) matches the context data of the content (e.g., construct a prompt based on the caption, context data, and a query requesting verification), or may perform verification by directly comparing the caption (or tag set) with the caption (or tag set) generated through the language model (22).

[0109] Heretofore, various embodiments of the tagging module / model operation method have been described with reference to FIGS. 9 to 11. As described above, the image analysis module (21) can be enhanced by using a set of content tags obtained through a language model (22) as training data. For example, a tag determiner (101) that determines whether a candidate tag is an important tag that matches the content can be constructed using the training data, and the tag determiner (101) can be added to the image analysis module (21). In addition, tagging of content can be performed without the help of a language model (22) through the enhanced image analysis module (21). In this case, the resources required for the tagging task can be significantly reduced, and the hallucination problem of the language model (22) can be completely eliminated.

[0110] Hereinafter, a method for creating a content collection according to some application examples of the present disclosure will be described with reference to FIGS. 12 and 13. While this method may be performed by a computing device (not shown) other than the tagging system (10), for ease of understanding, the following description assumes that it is performed by the tagging system (10). For reference, FIGS. 12 and 13 assume that the contents are from an OTT (Over The Top) service.

[0111] As illustrated in Figures 12 and 13, tags (i.e., tag sets) generated through the aforementioned tagging method can be utilized to automatically create a content collection. Here, a content collection refers to a collection of content with common characteristics (e.g., a collection of content with identical or similar topics, moods, characters, etc.).

[0112] For example, the tagging system (10) can automatically create a content collection (e.g., 128) by grouping pre-registered contents (e.g., 121 to 123) based on a common tag (e.g., 127). For example, the tagging system (10) can create (configure) a content collection (128) for a specific common tag (127) by grouping contents (e.g., 121, 122, etc.) having a specific common tag (127, 'a') among pre-registered contents (e.g., 121 to 123). At this time, the method of designating the specific common tag (127) may be any method.

[0113] As another example, the tagging system (10) can create a content collection based on the embedding similarity between contents (e.g., 121 to 123). Specifically, the tagging system (10) can create an embedding (e.g., an embedding vector) for contents (e.g., 121) based on a tag set (e.g., 124). For example, the tagging system (10) can create a content embedding by embedding the tag set (e.g., 124) using various word / text embedding techniques. Next, the tagging system (10) can create a content collection by grouping contents with an embedding similarity greater than a threshold. For example, the tagging system (10) can create a content collection by designating representative contents (e.g., popular contents, etc.) and grouping contents with an embedding similarity greater than a threshold with the representative contents. At this time, tags for the content collection can be selected from tags set for the representative contents.

[0114] As another example, the tagging system (10) may generate a content collection based on various combinations of the examples described above. For example, the tagging system (10) may generate a content collection (e.g., 128) by grouping contents with a common tag (e.g., 127) among contents with an embedding similarity exceeding a threshold value.

[0115] Fig. 13 illustrates an OTT service page (130). As illustrated in Fig. 13, various content collections (e.g., 128, 131) generated according to the above-described method can be provided to users through the OTT service page (130), and as a result, user satisfaction with the OTT service can be significantly improved.

[0116] So far, a method for creating a content collection according to some application examples of the present disclosure has been described with reference to FIGS. 12 and 13. Below, an exemplary computing device (140) capable of implementing a tagging system (10) according to some embodiments of the present disclosure will be described with reference to FIG. 14.

[0117] Figure 14 is an exemplary hardware configuration diagram showing a computing device (140).

[0118] As illustrated in FIG. 14, a computing device (140) may include one or more processors (141), a bus (143), a communication interface (144), a memory (142) for loading a computer program executed by the processor (141), and a storage (147) for storing a computer program (146). However, only components related to the embodiment of the present disclosure are illustrated in FIG. 14. Therefore, a person skilled in the art to which the present disclosure pertains will appreciate that other general components may be included in addition to the components illustrated in FIG. 14. That is, the computing device (140) may further include various components in addition to the components illustrated in FIG. 14. In addition, in some cases, the computing device (140) may be configured in a form in which some of the components illustrated in FIG. 14 are omitted. Hereinafter, each component of the computing device (140) will be described.

[0119] The processor (141) can control the overall operation of each component of the computing device (140). The processor (141) can be configured to include at least one of a CPU (Central Processing Unit), an MPU (Micro Processor Unit), an MCU (Micro Controller Unit), a GPU (Graphics Processing Unit), an NPU (Neural Processing Unit), or any other type of processor well known in the art of the present disclosure. In addition, the processor (141) can perform operations for at least one application or program for executing operations / methods according to embodiments of the present disclosure. The computing device (140) can include one or more processors.

[0120] Next, the memory (142) can store various data, commands, and / or information. The memory (142) can load a computer program (146) from the storage (147) to execute operations / methods according to embodiments of the present disclosure. The memory (142) may be implemented as a volatile memory such as RAM, but the technical scope of the present disclosure is not limited thereto.

[0121] Next, the bus (143) can provide a communication function between components of the computing device (140). The bus (143) can be implemented as various types of buses such as an address bus, a data bus, and a control bus.

[0122] Next, the communication interface (144) can support wired and wireless Internet communication of the computing device (140). Furthermore, the communication interface (144) can also support various communication methods other than Internet communication. To this end, the communication interface (144) can be configured to include a communication module well known in the technical field of the present disclosure.

[0123] Next, the storage (147) can non-temporarily store one or more computer programs (146). The storage (147) can be configured to include non-volatile memory such as Read Only Memory (ROM), Erasable Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), flash memory, a hard disk, a removable disk, or any form of computer-readable recording medium well known in the art to which the present disclosure pertains.

[0124] Next, the computer program (146) may include instructions that, when loaded into the memory (142), cause the processor (141) to perform operations / methods according to various embodiments of the present disclosure. That is, the processor (141) may perform operations / methods according to various embodiments of the present disclosure by executing the loaded instructions.

[0125] For example, the computer program (146) may include instructions that cause the image analysis module (21) to generate a candidate tag set including a plurality of candidate tags related to the content, construct a prompt based on the candidate tag set, context data for the content, and a query, and generate a tag set for the content based on a response of the language model (22) to the prompt.

[0126] As another example, the computer program (146) may include instructions to perform at least some of the steps / operations described with reference to FIGS. 1 through 13.

[0127] In the case illustrated, a tagging system (10) according to some embodiments of the present disclosure may be implemented via a computing device (140).

[0128] Meanwhile, in some embodiments, the computing device (140) illustrated in FIG. 14 may refer to a virtual machine implemented based on cloud technology. For example, the computing device (140) may be a virtual machine operating on one or more physical servers included in a server farm. In this case, at least some of the processor (141), memory (142), and storage (147) illustrated in FIG. 14 may be virtual hardware, and the communication interface (144) may also be implemented as a virtualized networking element, such as a virtual switch.

[0129] So far, an exemplary computing device (140) capable of implementing a tagging system (10) according to some embodiments of the present disclosure has been described with reference to FIG. 14.

[0130] Various embodiments of the present disclosure and effects according to the embodiments have been described with reference to FIGS. 1 through 14 so far. The effects according to the technical concept of the present disclosure are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

[0131] Furthermore, even though the above embodiments have described multiple components as being combined or operating in combination, the technical concept of the present disclosure is not necessarily limited to these embodiments. That is, within the scope of the technical concept of the present disclosure, all of the components may be selectively combined and operated one or more times.

[0132] The technical concepts of the present disclosure described so far can be implemented as computer-readable code on a computer-readable recording medium. A computer program recorded on the computer-readable recording medium can be transmitted to another computing device via a network such as the Internet, installed on said other computing device, and thus used on said other computing device.

[0133] Although the operations are depicted in a specific order in the drawings, it should not be understood that the operations must be performed in the specific order depicted, or in a sequential order, or that all depicted operations must be performed to achieve the desired result. In certain circumstances, multitasking and parallel processing may be advantageous. Although various embodiments of the present disclosure have been described above with reference to the attached drawings, those skilled in the art to which the present disclosure pertains will understand that the technical concepts of the present disclosure can be implemented in other specific forms without changing the technical concepts or essential characteristics thereof. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. The scope of protection of the present disclosure should be interpreted by the claims below, and all technical ideas within a scope equivalent thereto should be interpreted as being included in the scope of the technical ideas defined by the present disclosure.

Claims

1. A method performed by at least one computing device, A step of generating a candidate tag set including multiple candidate tags related to content through an image analysis module; A step of configuring a prompt based on the above candidate tag set, context data for the content, and a query statement; and A step of generating a set of tags for the content based on a response of a language model to the above prompt, How to tag your content.

2. In paragraph 1, The above content is video content including a first frame image and a second frame image, The above plurality of candidate tags include a first candidate tag extracted from the first frame image and a second candidate tag extracted from the second frame image through the image analysis module. How to tag your content.

3. In paragraph 1, The above context data includes metadata of the content, The above metadata includes at least one of the title, genre, and summary of the content. How to tag your content.

4. In paragraph 1, The above context data includes a script of the above content, How to tag your content.

5. In paragraph 1, The above image analysis module includes an object detection model and an action recognition model, The step of generating the above candidate tag set is: A step of detecting an object appearing in the content through the object detection model and generating a first candidate tag representing the detected object; and A step of recognizing the behavior of an object appearing in the content through the above behavior recognition model and generating a second candidate tag representing the recognized behavior, How to tag your content.

6. In paragraph 1, The above query statement includes a phrase requesting selection of important tags matching the content from the candidate tag set by referencing the context data. How to tag your content.

7. In paragraph 1, The steps to configure the above prompt are: A step of obtaining specific content tagged with a set of correct tags; A step of extracting candidate tags from the above specific content; A step of configuring a first prompt having the extracted candidate tag, context data of the specific content, and a query sentence as prompt elements; A step of evaluating a first response of the language model to the first prompt using the set of correct answer tags; A step of configuring a second prompt by changing at least one of the arrangement and content of the above prompt elements; Evaluating a second response of the language model to the second prompt using the set of correct answer tags; A step of generating a prompt template for the language model using the evaluation result of the first response and the evaluation result of the second response; and comprising a step of configuring the prompt using the above prompt template; How to tag your content.

8. In paragraph 1, The above language model is a model that has a limit on the maximum length of the input prompt, The steps to configure the above prompt are: A step of dividing the candidate tag set into a plurality of subsets based on the maximum length; and comprising a step of constructing a plurality of prompts based on each of the above subsets; How to tag your content.

9. In paragraph 1, The response of the above language model includes at least one tag, The steps for generating the above tag set are: A step of removing tags that do not exist in the candidate tag set from among the at least one tag and constructing the tag set using the remaining tags, How to tag your content.

10. In paragraph 1, A step of constructing a tag discriminator using the above candidate tag set and the above tag set, wherein the tag discriminator is a model that determines whether an input candidate tag is an important tag that matches the relevant content; Steps to obtain additional content; and A step of performing tagging on the additional content using the image analysis module to which the tag discriminator is added without the aid of the language model, How to tag your content.

11. In paragraph 10, The above tag determiner is configured to further receive image features of the relevant content extracted through the image analysis module and determine whether the input candidate tag is an important tag. How to tag your content.

12. In paragraph 1, The above set of tags is tagged to the above content, Further comprising a step of creating a content collection for the common tag by grouping other contents having a common tag with the content among a plurality of registered contents. How to tag your content.

13. One or more processors; and A memory storing a computer program executed by one or more processors, The above computer program: An operation of generating a set of candidate tags including multiple candidate tags related to content through an image analysis module; An operation of constructing a prompt based on the above candidate tag set, context data for the content, and a query statement; and comprising instructions for generating a set of tags for the content based on a response of a language model to the prompt; Tagging system.

14. Combined with a computing device, A step of generating a candidate tag set including multiple candidate tags related to content through an image analysis module; A step of configuring a prompt based on the above candidate tag set, context data for the content, and a query statement; and A computer-readable recording medium stored thereon to execute a step of generating a set of tags for the content based on a response of a language model to the above prompt. Computer program.

Images