Apparatus and method for manufacturing video content

Abstract

Disclosed are an apparatus and method for manufacturing video content corresponding to a user's intention with high accuracy. According to an aspect of the present embodiment, provided are the apparatus and method for manufacturing video content matching the user's intention with high accuracy by analyzing a query of the user, determining what characteristics the query has, and performing a search according to a result of the determination.

Description

Device and method for producing video content

[0001] The present invention relates to a device and method for producing video content that matches a user's intention with high accuracy.

[0002] This invention is a technology developed through the Seoul Economic Promotion Agency’s 2024 Creative Industry Technology Commercialization Support Project (CC240014) “Development of a video editing solution that enables cut editing and background music creation with just a click for the digitally disadvantaged.”

[0003] The content described in this section merely provides background information for the present embodiment and does not constitute prior art.

[0004] The field of artificial intelligence (AI) has experienced a new heyday over the past several years, fueled by advances in deep learning technology and the rapid growth of research on large-scale language models. In particular, generative AI models have significantly advanced the ability to process and generate natural language. Generative AI models are gaining attention as they expand into diverse applications, including conversational chatbots, automatic summarization systems, and creative text generation tools.

[0005] Generative AI models are defined as technologies that aim to enable machines to go beyond simply analyzing input text or images, but instead directly generate new data. While this concept has existed for a long time, it has recently gained significant attention, fueled by advances in deep learning and large-scale language models (LLMs). Generative AI is a form of AI that can generate new content, such as text, images, audio, or video, based on LLMs. It is being applied in diverse fields, from everyday conversations to finance, healthcare, education, and entertainment.

[0006] At this time, LLM's core technology is the Transformer architecture. Unlike existing language model technologies that process sentences sequentially, Transformers process entire sentences at once, allowing them to more effectively learn the relationships between words. This allows generative AI to go beyond simple sentence generation and perform a variety of language-related tasks, including complex question-answering, text analysis, creative writing, translation, and summarization.

[0007] While LLMs, or Large Multimodal Models (LMMs), can process diverse data formats, including text, images, voice, and video, they are limited to static data, processing not only language-related information but also non-verbal information such as images, sounds, and videos. Existing LLMs and LMMs are limited in their ability to respond to real-time changing environments or directly interact with the physical world. This makes it difficult to analyze current situations and determine immediate actions accordingly. Furthermore, LLM-based generative AI models only process information necessary for individual tasks, limiting their comprehensive understanding and prediction of the entire environment. Furthermore, they face structural limitations that make it difficult to immediately reflect up-to-date information. Consequently, LLM-based generative AI models may generate answers that conflict with current information (e.g., when faced with a lack of factual information). Furthermore, because they rely on previously learned knowledge, they may lack a high level of understanding of new problems or domains.

[0008] Various solutions are being explored, including domain-specific fine-tuning of LLMs and RAGs, which utilize customized datasets for each company to enhance reliability. Currently, many companies are leveraging LLM and RAG technologies to introduce AI systems in various fields, such as information retrieval, content summarization, and production. However, these systems suffer from limitations such as data bias, hallucination issues, difficulties in real-time data processing, and a lack of domain-specific knowledge.

[0009] In particular, there are frequent cases where random answers are provided without considering the characteristics of the user's question. For example, even if the user asks a question about abstract and general things, such as the properties of a specific object, the conventional RAG model may generate and provide only answers that precisely match the words in the question. Conversely, even if the user inquires about specific things related to a specific object, such as a phenomenon occurring in the object or its internal structure, the conventional RAG model may provide abstract answers about the specific object. As a result, the user experiences the inconvenience of not receiving an answer that properly matches the intent of the inquiry or having to wait a considerable amount of time to receive it.

[0010] One embodiment of the present invention aims to provide a device and method for producing video content that matches a user's intention with high accuracy by analyzing a user's query to determine what characteristics it has and performing a search in accordance with the determination result.

[0011] One embodiment of the present invention aims to provide a video indexing device and method that significantly improves indexing performance while reducing the amount of computation required to index a video.

[0012] In addition, one embodiment of the present invention provides a method for producing video content using a RAG-based generative AI model capable of providing response data that enables the generation of video content corresponding to the user query through a large-scale language model based on the searched clip video and the user query by decomposing a user query into one or more sub-queries to understand the user intent using a multi-agent system and a RAG system, searching for raw video data input by the user in units of clips for each sub-query, and providing a computing device that executes the method.

[0013] According to one aspect of the present embodiment, a video content production device is provided, comprising: a user interface module that receives raw video content and a user query from a user terminal and provides a response corresponding to the received user query to the user terminal; a video analysis and storage module that divides raw video data into a plurality of clip images and analyzes them, and stores the results of each video analysis; an agent execution module that analyzes a user query and decomposes it into one or more sub-queries, specifically plans and organizes a task for resolving a user request into sub-tasks, and executes the task to search for a clip using the sub-query; and a RAG execution module that searches for a scene corresponding to the sub-query generated by the agent execution module from the video analysis and storage module, processes the searched information, and inputs the searched information into a generative AI model to generate final response data.

[0014] According to one aspect of the present embodiment, the RAG execution module is characterized by including a search module that searches for a scene corresponding to a sub-query generated by the agent execution module from the video analysis and storage module, an augmentation module that processes information searched by the search module into an augmented context and inputs it into a generative AI model, and a generation module that generates final response data using the augmented context.

[0015] According to one aspect of the present embodiment, the search module is characterized in that it analyzes a query or subquery entered by a user and determines what characteristics it has.

[0016] According to one aspect of the present embodiment, the search module is characterized in that it divides the query characteristics into first characteristics and second characteristics.

[0017] According to one aspect of the present embodiment, the first characteristic is characterized in that it indicates a relatively comprehensive and vague degree, such as a property, characteristic, or aspect of a specific object.

[0018] According to one aspect of the present embodiment, the second characteristic is characterized by indicating a relatively specific and clear degree, such as a phenomenon occurring in a specific object, an internal configuration, structure or component of a specific object.

[0019] According to one aspect of the present embodiment, the search module is characterized in that it analyzes a query or subquery entered by a user and determines to what extent the queries possess the first characteristic and the second characteristic, respectively.

[0020] According to one aspect of the present embodiment, the search module is characterized in that it assigns weights to search results according to each characteristic value.

[0021] According to one aspect of the present embodiment, the search module is characterized in that it provides search results in order of search results that best match the query based on weighted results.

[0022] According to one aspect of the present embodiment, a method for producing video content by a video content production device is provided, the method comprising: a receiving process for receiving raw video data and a user query including a user request; a storing process for dividing the raw video data into a plurality of clip images and analyzing them, and storing the video analysis results including each clip image, a thumbnail image representing each clip image, and augmented summary data from each clip image in a video analysis / storage module; an execution process for decomposing a user query into one or more subqueries through semantic analysis based on natural language processing using a multi-agent system, and executing one or more agents to execute the user request for each decomposed subquery; and a generating process for calling a RAG system for each sub-task, searching for a clip image in the video analysis results corresponding to the user request for each subquery, and generating response data including a video editing code using a pre-trained artificial intelligence model based on the searched clip image and the user query.

[0023] As described above, according to one aspect of the present embodiment, by analyzing a user's query to determine what characteristics it has and performing a search in accordance with the determination result, video content that matches the user's intention with high accuracy can be produced, thereby improving user satisfaction.

[0024] According to one aspect of the present embodiment, when raw video data and a user query are received from a user, the multi-agent system and the RAG system are used to search for the video analysis results of the raw video data stored in the database, and response data is generated based on the searched results through a large-scale language model, thereby generating response data that reflects the user's intent, thereby having the advantage of improving the accuracy and reliability of the response data.

[0025] In addition, the present invention has the advantage of being able to produce more accurate and richer video content by post-processing RAG-based response data, and of overcoming limitations such as difficulty in reflecting the latest information and errors in factual relationships (hallucinations) by complementing the shortcomings of existing LLM.

[0026] In addition, according to one aspect of the present embodiment, there is an advantage of significantly improving indexing performance while reducing the amount of computation required to index a video.

[0027] Figure 1 is a block diagram of a video content production device according to one embodiment of the present invention.

[0028] FIG. 2 is a drawing illustrating the configuration of a processor in a video content production device according to one embodiment of the present invention.

[0029] FIG. 3 is an exemplary diagram for implementing a multi-agent system according to one embodiment of the present invention.

[0030] FIG. 4 is an exemplary diagram illustrating an agent execution module and a RAG execution module according to one embodiment of the present invention.

[0031] FIG. 5 is a diagram illustrating the configuration of a video analysis / storage module according to one embodiment of the present invention.

[0032] FIG. 6 is a drawing illustrating an example of a clip extracted by a clip extraction unit according to one embodiment of the present invention.

[0033] FIG. 7 is a diagram illustrating information indexed by an indexing unit according to one embodiment of the present invention.

[0034] FIG. 8 is a drawing illustrating an example of clips being grouped by a grouping unit according to one embodiment of the present invention.

[0035] FIGS. 9 and 10 are diagrams illustrating a process of generating an image by a grouping unit according to one embodiment of the present invention.

[0036] FIG. 11 is a flowchart illustrating a method for a video content production device according to one embodiment of the present invention to produce video content using a RAG-based generative AI model.

[0037] FIG. 12 is an exemplary diagram illustrating a method for producing video content using a RAG-based generative AI model by a video content production device according to one embodiment of the present invention.

[0038] FIG. 13 is an exemplary diagram illustrating a clip video recommendation process by a RAG system in a video content production device according to one embodiment of the present invention.

[0039] FIG. 14 is a flowchart illustrating a method for a video analysis / storage module to index a video according to one embodiment of the present invention.

[0040] The present invention is susceptible to various modifications and embodiments. Specific embodiments are illustrated and described in detail in the drawings. However, this is not intended to limit the present invention to specific embodiments, but rather to encompass all modifications, equivalents, and alternatives falling within the spirit and technical scope of the present invention. Throughout the description of each drawing, similar reference numerals have been used to designate similar components.

[0041] Terms such as first, second, A, and B may be used to describe various components, but these components should not be limited by these terms. These terms are used solely to distinguish one component from another. For example, without departing from the scope of the present invention, a first component may be referred to as a second component, and similarly, a second component may also be referred to as a first component. The term "and / or" includes a combination of multiple related items described herein or any of multiple related items described herein.

[0042] When a component is referred to as being "connected" or "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but that there may be other components intervening. Conversely, when a component is referred to as being "directly connected" or "connected" to another component, it should be understood that there are no other components intervening.

[0043] The terminology used in this application is solely for the purpose of describing specific embodiments and is not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly dictates otherwise. It should be understood that terms such as "comprise" or "have" in this application do not preclude the presence or possibility of addition of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification.

[0044] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs.

[0045] Terms defined in commonly used dictionaries should be interpreted to have a meaning consistent with their meaning in the context of the relevant technology, and will not be interpreted in an idealized or overly formal sense unless expressly defined in this application.

[0046] In addition, each configuration, process, procedure or method included in each embodiment of the present invention may be shared within a scope that is not technically inconsistent with each other.

[0047] Figure 1 is a block diagram of a video content production device according to one embodiment of the present invention.

[0048] A video content production device (10) according to one embodiment of the present disclosure is a system for producing video content using a RAG-based generative AI model, and may include a server of any type and / or a device of any type.

[0049] Specifically, the video content production device (10) may be a hardware device or a part of a hardware device that performs comprehensive processing and calculation of data, or may be a software-based computing environment connected to a communication network. For example, the video content production device (10) may be a server that performs intensive data processing functions and shares resources, or may be a client that shares resources through interaction with a server. In addition, the video content production device (10) may be a cloud system in which a plurality of servers and clients interact to comprehensively process data. Since the above description is only one example related to the type of the video content production device (10), the type of the video content production device (10) may be configured in various ways within a range that can be understood by those skilled in the art based on the contents of the present disclosure.

[0050] Referring to FIG. 1, a video content production device (10) according to an embodiment of the present disclosure may include a processor (11), a memory (12), and a network unit (13). However, FIG. 1 is merely an example, and thus the video content production device (10) may include other components for implementing a computing environment. In addition, only some of the components disclosed in FIG. 1 may be included in the video content production device (10).

[0051] The processor (11) may be understood as a configuration unit including hardware and / or software for performing computing operations. For example, the processor (11) may read a computer program to perform data processing for machine learning. The processor (11) may process computational processes such as processing input data for machine learning, feature extraction for machine learning, and error calculation based on backpropagation. The processor (11) for performing such data processing may include a central processing unit (CPU), a general purpose graphics processing unit (GPU), a tensor processing unit (TPU), an application specific integrated circuit (ASICc), or a field programmable gate array (FPGA). The type of the processor (11) described above is only one example, and thus, the type of the processor (11) may be configured in various ways within a range understandable to those skilled in the art based on the contents of the present disclosure.

[0052] The processor (11) can learn one or more artificial intelligence models using an architecture that structures the collected dataset and learning algorithm into layers, and can fine-tune the learned artificial intelligence model by applying a specific dataset. Here, fine-tuning means updating an existing learned artificial intelligence model by modifying the architecture of the existing learned artificial intelligence model to suit the purpose.

[0053] Additionally, the processor (11) can evaluate the performance of each artificial intelligence model by dividing the collected data set into learning data, test data, and verification data types and executing a learning algorithm. At this time, the learned artificial intelligence model can be divided into various types depending on the learning algorithm.

[0054] Generative AI models leverage AI technology to automatically generate videos based on other forms of data, such as text, images, or videos. Generative AI models can leverage AI video production tools in a variety of ways, such as transforming raw video data into new video content that reflects user intent. These generative AI models include LLMs and multimodal LLMs, but can also include image generation models, audio generation models, or other generative models.

[0055] The processor (11) implements a generative AI model using agent-based Augmented Search Generation (RAG) technology, utilizes an LLM-based agent to automate the search and answer generation process, and enables more complex tasks by combining various tools as needed. Specifically, during the search process, the processor (11) analyzes the user's query to determine its characteristics and performs the search in accordance with the determined results. Accordingly, the processor (11) can generate and provide an answer that best matches the user's query intent.

[0056] In addition to the examples described above, the types of data used to train an AI model and the output of the AI ​​model can be configured in various ways within a range understandable to those skilled in the art based on the contents of this disclosure. Furthermore, at least one AI model may be used, and multiple AI models may be integrated into a single network, partially share a network, or be implemented as separate, independent networks.

[0057] The memory (12) may be understood as a configuration unit including hardware and / or software for storing and managing data processed in the video content production device (10). That is, the memory (12) may store any type of data generated or determined by the processor (11) and any type of data received by the network unit (13). For example, the memory (12) may include at least one type of storage medium among a flash memory type, a hard disk type, a multimedia card micro type, a card type memory, a random access memory (RAM), a static random access memory (SRAM), a read only memory (ROM), an electrically erasable programmable read only memory (EEPROM), a programmable read only memory (PROM), a magnetic memory, a magnetic disk, or an optical disk. In addition, the memory (12) may also include a database system that controls and manages data in a predetermined system. The type of memory (12) described above is only one example, and thus the type of memory (12) can be configured in various ways within a range understandable to those skilled in the art based on the contents of the present disclosure.

[0058] The memory (12) can structure and organize and manage data, combinations of data, and program codes executable by the processor (11) required for the processor (11) to perform operations. For example, the memory (12) can store various types of data via the network unit (13) described later. The memory (12) can store program codes that operate an artificial intelligence model to perform learning, program codes that operate an artificial intelligence model to receive data and perform inference according to the purpose of use of the video content production device (10), and processed data generated as the program codes are executed. In addition, the memory (12) can store at least one database in a separate space or in the same space. The database can be stored in the memory (12) or in a storage area managed by the processor (11). That is, the database can be stored in the memory (12), stored in a memory different from the memory (12), or stored in an external memory.

[0059] The network unit (13) can be understood as a component that transmits and receives data via any type of known wired and wireless communication system. For example, the network unit (13) can perform data transmission and reception using a wired and wireless communication system such as a local area network (LAN), wideband code division multiple access (WCDMA), long term evolution (LTE), wireless broadband internet (WIBRO), fifth generation mobile communication (5G), ultra wide-band, Zigbee, radio frequency (RF) communication, wireless LAN, wireless fidelity, near field communication (NFC), or Bluetooth. Since the above-described communication systems are only examples, the wired and wireless communication system for data transmission and reception of the network unit (13) can be applied in various ways other than the above-described examples.

[0060] The network unit (13) performs wired or wireless communication with any system or any client, etc., and can receive data necessary for the processor (11) to perform calculations. In addition, the network unit (13) performs wired or wireless communication with any system or any client, etc., and can transmit data generated through calculations of the processor (11). The network unit (13) performs communication with the aforementioned database, server, or computing device, etc., and can transmit output data of the artificial intelligence model and intermediate data or processed data derived from the calculation process of the processor (11).

[0061] FIG. 2 is a diagram illustrating a configuration of a processor in a video content production device according to one embodiment of the present invention, FIG. 3 is an exemplary diagram for implementing a multi-agent system according to one embodiment of the present invention, and FIG. 4 is an exemplary diagram for explaining an agent execution module and a RAG execution module according to one embodiment of the present invention.

[0062] Referring to FIG. 2, a processor (11) according to one embodiment of the present invention includes, but is not limited to, a user interface module (110), an agent execution module (120), a video analysis and storage module (130), and a RAG execution module (140).

[0063] The processor (11) can be implemented by modularizing various functions, for example, user interface functions, agent execution functions, RAG execution functions, and video analysis and storage functions into modules.

[0064] The user interface module (110) receives raw video content and user queries from a user terminal, and provides a response corresponding to the received user query to the user terminal. At this time, the user interface module (110) can receive the user's voice using a microphone and convert it into an input signal corresponding to the user query. In this case, the video content production device (10) can convert the user's speech data into natural language query text by utilizing voice recognition technology such as STT (Speech-To-Text).

[0065] The agent execution module (120) can be implemented as a multi-agent system. A multi-agent system consists of two or more interacting AI agents, each performing tasks independently or cooperatively. An AI agent is an LLM-based entity that repeatedly develops and executes plans to achieve a goal. Each AI agent is assigned a persona to perform a specific role, which defines the agent's description and available tools.

[0066] This multi-agent system includes a planning agent group (121) and a function agent group (122). The planning agent group (121) plans the sequence of actions to be executed to respond to user queries by understanding user intent, and is responsible for planning and processing to guide users to the information or services required for specific requests. The function agent group (122) performs defined functions based on the results of analyzing user queries. The multi-agent system can be designed modularly to facilitate expansion while minimizing conflicts between independent agents, and the modularly designed groups can provide flexibility for customizing to various domains by easily integrating new agents or removing unnecessary agents.

[0067] The planning agent group (121) can perform a query analysis function to analyze and regenerate an input user query to understand the user's intention, a plan concretization function to specifically plan and organize sub-tasks for resolving a user request, a collation and processing function to organize and collate the results of completed sub-tasks, a task status tracking function to track and analyze the progress of sub-tasks, and a path setting function to determine the next step based on task status analysis to proceed with a plan to resolve a user query.

[0068] The functional agent group (122) is composed of multiple agents for executing each function of: generating subqueries that generate subqueries based on the analysis results of raw video data and query analysis; executing external APIs that execute external APIs (such as external search engines) in addition to defined functions; video curation that performs the searched video content creation step; generating prompts that automatically generate prompts that are input to a generative AI model; and executing and organizing RAG that searches for clips using subqueries and creates formats such as JSON files, XML, and TAML for video curation. The agent for generating subqueries can be created by considering the indexing values ​​(of clips or clip groups) processed by the video analysis and storage module (130) described below for the raw video data when generating subqueries based on the analysis results of raw video data and query analysis results. Accordingly, the functional agent group (122) can provide more accurate analysis or search results, etc. when processing the raw video data according to the user's query.

[0069] In this way, the agent execution module (120) can resolve a user request by decomposing a task corresponding to a user request into one or more subqueries (subtasks), assigning roles, performing collaborative execution, and integrating results.

[0070] The video analysis and storage module (130) divides raw video data into multiple clip videos and analyzes them, and stores the video analysis results, including each clip video and a thumbnail image representing each clip video, in a database. At this time, the video analysis and storage module (130) can create and store multiple thumbnail images for one clip video when the clip video exceeds a preset time (for example, about 4 to 5 seconds). In addition, the video analysis and storage module (130) can analyze the raw video data and then reconstruct it into a semantic clip video and enhance summary data in each clip video. The video analysis and storage module (130) can convert voice in the video data into text by utilizing various artificial intelligence models such as Whisper AI, Maestra AI, Rask AI, or GitMind, and the text thus converted can be utilized for various purposes such as subtitle creation, content summary, and search function enhancement.

[0071] The video analysis and storage module (130) can provide augmented information, including thumbnail images and summary data corresponding to each clip video, as a video analysis result. Furthermore, since the video analysis and storage module (130) analyzes videos using thumbnail images of clip videos, it has the advantage of providing low-cost yet fast analysis results.

[0072] Furthermore, the video analysis and storage module (130) performs indexing on the extracted clips or on clip groups additionally grouped using the extracted clips. The video analysis and storage module (130) can significantly reduce the amount of data to be indexed compared to the past by performing indexing on the clips or clip groups extracted from the video according to the method described below. Accordingly, the video analysis and storage module (130) can quickly index the input video with a smaller amount of computation. This can bring great benefits to external devices that analyze or search videos. These external devices use artificial intelligence learning models such as LLM (Large Language Model) when analyzing or searching videos. However, these artificial intelligence learning models have the problem that when the amount of data to be processed increases, the accuracy of the inference value output significantly decreases and the time required to output the inference value also increases significantly. Accordingly, when analyzing or searching a video as a whole, it takes a long time and the accuracy of the results also significantly decreases. On the other hand, when indexing is performed through the video analysis and storage module (130), analysis or search can be performed using indexed clips or clip groups rather than the entire video, so that an external device can infer results that are both relatively fast and highly accurate.

[0073] Furthermore, the video analysis and storage module (130) performs indexing not only on extracted clips but also on clip groups, thereby enabling relatively fast and accurate processing when analyzing or searching video content. After extracting clips, the video analysis and storage module (130) additionally groups matching clips according to each type. For example, if a video to be indexed is a baseball play video, the video analysis and storage module (130) can extract clips in the video such as clips in which the batter hits the ball, clips in which the batter misses the ball, clips in which the defender catches the ball, and clips in which the defender does not catch the ball. Furthermore, the video analysis and storage module (130) can group the extracted clips into a clip group for hits, a clip group for home runs, a clip group for strikeouts, a clip group for increasing the out count, etc., and can further group the grouped clip groups into a clip group for scoring, a clip group for striking outs, etc. Accordingly, the video analysis and storage module (130) can extract clips or clip groups with various topics or information about the video, and thus other devices can more accurately analyze or search the video content using the indexing information. A more detailed description of the video analysis and storage module (130) will be described later with reference to FIGS. 5 to 10.

[0074] The RAG execution module (140) retrieves external data during the text generation process and integrates the retrieved information with pre-learned content to generate more accurate and rich content. The RAG execution module (140) can be implemented as an agent-based RAG system, which can respond to various question types and effectively generate answers even when domain-specific knowledge is lacking.

[0075] The RAG execution module (140) includes a search module (141), an augmentation module (142), and a generation module (143).

[0076] The search module (141) searches for a scene corresponding to a subquery generated by the function agent module (122) from the video analysis and storage module (130). The search module (141) searches for a scene corresponding to the subquery in units of clip images or clip groups, including augmented information including thumbnail images and summary data, based on the video analysis results stored by the video analysis and storage module (130), and generates the search results as response data in text form and provides the result to the augmentation module (142).

[0077] The search module (141) analyzes the query or subquery input by the user to determine what characteristics it has when searching for a scene corresponding to the subquery from the video analysis and storage module (130). The characteristics of the query can be divided into first and second characteristics. The first characteristic refers to a relatively comprehensive and vague degree, such as a property, characteristic, aspect, qualitative characteristic, or abstract characteristic of a specific object, and the second characteristic refers to a relatively specific and clear degree, such as a phenomenon, internal composition, structure, component, quantitative characteristic, or concrete characteristic occurring in a specific object. The search module (141) analyzes the query or subquery input by the user to determine to what extent the corresponding queries have the corresponding characteristics for each of the first and second characteristics. The search module (141) can derive a result by reflecting the determination result when performing a search in units of clip videos or groups of clips corresponding to the query. For example, the search module (141) can perform a search by considering both the first characteristic and the second characteristic, and can assign weights to the search results according to each characteristic value. By assigning weights to the search results, the search module (141) can provide search results in the order of search results that best match the query (search results that are determined to have a high similarity due to a high weight) based on the weighted results. In this way, the search module (141) can search for clips or groups of clips that match the user's intention with high accuracy by judging the characteristics of the query and assigning weights using the judgment results.

[0078] Furthermore, the search module (141) can find appropriate data from a large document collection or database. If a generative AI model relies solely on learned parameters, the knowledge may be less up-to-date or accurate. To prevent this, the augmentation module (142) can serve as a gateway for obtaining necessary information from external knowledge sources, and the search module (141) uses embedding-based search (Vector Similarity Search) technology to convert natural language sentences or words into vector form, and then measures the similarity (such as cosine similarity) between these vectors to search for related documents. These embeddings can be obtained from various models such as BERT or Sentence Transformers, and large-scale vector data can be efficiently searched using vector indexing libraries such as FAISS or Annoy.

[0079] The augmentation module (142) processes the information searched by the search module (141) into an augmented context and inputs it into the generative AI model (200). The augmented context may be composed of the results of summarizing, extracting, and filtering the searched information or a portion of the searched information so that the generative AI model can refer to it when generating an answer.

[0080] The response quality of a generative AI model can vary significantly depending on the format in which the context is referenced and how the query or answer requirements are set. Therefore, the RAG execution module (140) utilizes prompt design (prompt engineering) technology to connect the query and context as naturally as possible and explicitly provide the information the model should reference, thereby generating answers that more accurately reflect the actual user intent.

[0081] The generation module (143) generates final response data using the augmented context. Various large-scale language models (200), such as the GPT series, T5 series, or BERT-based generative models, can be utilized, and recently, cases of using fine-tuned dedicated models or the latest APIs are also increasing. The generation module (143) generates an answer by synthesizing the information acquired by the search module (141) and the augmentation module (142) and its own learned knowledge, while maintaining factuality and contextual consistency as much as possible. The response data generated by the generation module (143) may be delivered to the user terminal after a separate post-processing process or may be linked to other modules within the system.

[0082] As illustrated in FIG. 3, when a user query is "Summarize how to make egg bread. Focus on cooking scenes, not eating scenes," the agent execution module (120) decomposes the user request into one or more sub-queries so that the user can search more specifically. At this time, the agent execution module (120) can decompose the user query into one or more sub-queries based on the indexed information of each clip, and can quickly and easily search for clip videos corresponding to each sub-query by searching the indexed information within the database (135) of the video analysis and storage module (130).

[0083] For example, a subquery can be broken down into the following scenes: preparing ingredients (preparing the necessary ingredients), beating eggs (breaking and mixing eggs), and soaking bread (dipping bread in egg water).

[0084] In this way, the RAG execution module (140) searches for a scene corresponding to a subquery in units of clip images including augmented information including thumbnail images and summary data based on the video analysis results stored by the video analysis and storage module (130), and generates and provides the searched results in response data in text form.

[0085] At this time, the RAG execution module (140) can generate response data including indexed information of the clip video (e.g., id: "c14"), summary text and text vector (e.g., "Soak the bread in egg water with both hands"), script text and text vector (e.g., "Oh, now all you have to do is soak the bread and all the annoying work is over"), and image frames and image vectors.

[0086] FIG. 5 is a diagram illustrating the configuration of a video analysis / storage module according to one embodiment of the present invention.

[0087] Referring to FIG. 5, a video analysis / storage module (130) according to one embodiment of the present invention includes a communication unit (510), a clip extraction unit (520), an indexing unit (530), a grouping unit (540), a memory (550), and a post-processing unit (560).

[0088] The video analysis / storage module (130) indexes the input video, enabling relatively fast and accurate processing when analyzing or searching the video content. The video analysis / storage module (130) stores the video analysis results, including each clip video or each clip group and thumbnail images representing each clip or clip group.

[0089] The communication unit (510) receives a video for extracting each clip or group of clips from an external source. The communication unit (510) performs wired or wireless communication to receive a video from an external device.

[0090] In some cases, the communication unit (510) may additionally receive grouping conditions from an external device when grouping each clip. As described below, the communication unit (510) may additionally receive conditions for grouping a first clip group into a second clip group from an external device.

[0091] The clip extraction unit (520) extracts each clip separately from the input video.

[0092] The clip extraction unit (520) can divide the received video into each frame. The clip extraction unit (520) determines whether a change has occurred in the pixel value or HSV (hue, saturation, brightness) within each frame beyond a preset standard. The clip extraction unit (520) may determine the pixel value or HSV of the entire frame, or, for more detailed and accurate determination, divide the entire frame into multiple sections and determine the pixel value or HSV for each section. If a difference has occurred in the pixel value or HSV of the entire preceding and succeeding frames or in each section beyond a preset standard, the clip extraction unit (520) determines that the clip has changed between the analyzed preceding and succeeding frames. The clip extraction unit (520) divides the frame following the last frame in which the clip has been previously determined to have changed into one clip to the last frame in which the clip has been determined to have changed at the current point in time, and divides the next clip under the same conditions. The clip extraction unit (520) extracts the video input in this manner by dividing it into one or more clips.

[0093] Alternatively, the clip extraction unit (520) distinguishes and extracts clips based on edge changes of objects within each frame. The clip extraction unit (520) converts each frame to gray scale, removes basic noise, and then extracts the edges of objects within each frame using various edge detection methods (e.g., Canny Edge Detector, Sobel Filter, or Laplacian Filter). The edges of the objects are extracted in the form of binary images. The clip extraction unit (520) extracts the edges of objects for each section or each unit pixel between each frame, and calculates their average values. Similarly, the clip extraction unit (520) accumulates the above-described average values ​​between the preceding and succeeding frames, and determines whether the accumulated value of the average value has changed by more than a preset reference value. The clip extraction unit (520) extracts clips based on edge changes of objects based on the above-described method.

[0094] Alternatively, the clip extraction unit (520) may operate in parallel or selectively with the aforementioned method, and extracts clips based on changes in the conversation or non-conversation sounds within the video. The clip extraction unit (520) recognizes the conversation (Voice) or non-conversation sounds (Sound) within the video. When recognizing the conversation (Voice), the clip extraction unit (520) may use a method for converting conversation or voice into text, such as Whisper AI, Maestra AI, Rask AI, or GitMind. Typically, the Whisper artificial intelligence model receives voice data as input, analyzes frequency, intensity, or temporal change in the voice data, and extracts acoustic features. Thereafter, the Whisper artificial intelligence model recognizes features such as language or pronunciation from the extracted acoustic features and converts them into text. The clip extraction unit (520) recognizes changes in the conversation content using the converted text. Changes in the content of a conversation may occur when a sentence ends with a final ending such as '다' or '까' (in languages ​​with a final ending), or when the speaker's breathing is interrupted. As described above, when a sentence ends using text or the speaker's breathing is interrupted, the clip extraction unit (520) considers that a change has occurred in the content of the conversation and can extract the corresponding point in time by dividing it into cuts. The clip extraction unit (520) can use a method for recognizing sound events, such as a sound event detection (SED) model, when recognizing sound. The sound event detection model recognizes the occurrence, start, and end points of specific or various sound events within the input data. The clip extraction unit (520) recognizes the occurrence / start of various sound events within the video, and when a specific sound event ends depending on the type of sound event, the clip extraction unit (520) can recognize that a change has occurred in the sound.The clip extraction unit (520) can extract the corresponding point in time by dividing it into cuts.

[0095] Clips extracted by the clip extraction unit (520) are illustrated in FIG. 6.

[0096] FIG. 6 is a drawing illustrating an example of a clip extracted by a clip extraction unit according to one embodiment of the present invention.

[0097] Referring to FIG. 6, the clip extraction unit (520) distinguishes clips as the time before and after a point in time when a change occurs in the pixel value or HSV of a frame within a video that exceeds a preset standard, or as the time before and after a point in time when a change occurs in dialogue or sound. Accordingly, the clip extraction unit (520) extracts at least one clip from the video.

[0098] Referring back to FIG. 5, the clip extraction unit (520) can extract a script for each extracted clip. The clip extraction unit (520) can convert the extracted clips (e.g., files in mp4 format, etc.) into audio files (e.g., files in mp3 format, etc.) and extract text (script) from the audio files. The clip extraction unit (520) can utilize the Whisper artificial intelligence model, etc., to extract the text.

[0099] The indexing unit (530) performs indexing on each clip extracted from the clip extraction unit (520) and, further, on clip groups grouped in the grouping unit (540) described later.

[0100] The indexing unit (530) performs indexing not only on each clip extracted from the clip extraction unit (520), but also on clip groups grouped by the grouping unit (540) described below. The indexing unit (530) performs indexing so that an external device can find and analyze or search each clip from the video received by the communication unit (510). However, unlike the conventional method, the indexing unit (530) performs indexing on each clip or clip group as illustrated in FIG. 7.

[0101] FIG. 7 is a diagram illustrating information indexed by an indexing unit according to one embodiment of the present invention.

[0102] The indexing unit (530) indexes OCR (text), Vision (visual elements), Voice (conversation or voice), Sound (auditory elements other than conversation), and Context (context, context) information for each clip or group of clips to be indexed. Here, Context refers to information indicating context, and goes beyond individual analysis of the visual and auditory elements of a specific clip to comprehensively analyze the relationship with the clips before and after the clip, or the semantic association between objects, people, and actions included in the clip, which is semantic information generated. For example, if the clip of 'a pitcher throwing a ball' and the clip of 'a batter hitting a ball' are consecutive, the context of these two clips can be defined as a 'pitching event'. In the past, when indexing a video, rather than extracting clips within the video and indexing them like the video analysis / storage module (130), indexing was performed on a certain section or the entire video content. Accordingly, in the past, when indexing a video, it was sufficient to index only visual elements (Voice) and conversation or voice information (Voice), and when attempting to index more information, there was the inconvenience of a significant increase in the amount of data to be processed.

[0103] On the other hand, the indexing unit (530) performs indexing using the five elements described above for each clip or group of clips. As described above, since the clip extraction unit (520) extracts clips from within a video, if only visual elements and dialogue or audio information are used for the extracted clips as in the past, indexing may be performed with considerable inaccuracy. Considering this, the indexing unit (530) adds text (OCR), auditory elements other than dialogue (Sound), and context (Context) in addition to the above-described content. By adding text to the information to be indexed, the indexing unit (530) can more accurately extract clips from videos provided in the form of text together with dialogue or audio. By adding auditory elements other than dialogue to the information to be indexed, the indexing unit (530) can more clearly recognize scene changes or mood changes within the video. In addition, the indexing unit (530) can improve indexing accuracy that is lower when performing indexing based solely on visual elements by adding context to the information to be indexed.

[0104] The indexing unit (530) indexes caption information for text (OCR) and dialogue or voice, and both caption information and vector information for visual elements (Vision), auditory elements (Sound), and context. For visual elements, auditory elements, and context that are relatively more complex and contain more elements, the indexing unit (530) comprehensively indexes vector information for the corresponding clips, etc., and specifically indexes caption information.

[0105] When indexing a clip, the indexing unit (530) can adjust the weight of information to be indexed differently by considering the factors used when extracting the clip by the clip extraction unit (520). For example, if the clip extraction unit (520) extracts the clip based on the pixel value or HSV or edge change of the object within each frame, the indexing unit (530) can perform indexing with a relatively greater weight on text (OCR), visual elements (Vision) or context (Context). Conversely, if the clip extraction unit (520) extracts the clip based on the content of the conversation or changes in sounds other than the conversation, the indexing unit (530) can perform indexing with a relatively greater weight on the conversation, voice information or auditory elements (Sound).

[0106] Furthermore, the indexing unit (530) additionally performs indexing on the (first and / or second) clip groups grouped by the grouping unit (540) described below. While video viewers or video editors may search for or analyze individual clips when analyzing or searching for videos, they may also frequently search for entire groups of clips with similar content. In this case, if indexing is not performed on the clip groups themselves, viewers may experience significant inconvenience because they can only search or analyze each clip individually. Therefore, the indexing unit (530) additionally performs indexing on the clip groups grouped by the grouping unit (540) described below. The indexing unit (530) performs indexing on clip groups using the same elements as clips. Since the indexing unit (530) performs indexing on clip groups, viewers can search or analyze not only clips but also groups of clips of their choice.

[0107] The grouping unit (540) groups clips within a certain range into first clip groups based on contextual similarity using the indexing information of each clip. Furthermore, the grouping unit (540) can additionally group first clip groups having similar content by determining the similarity between clips based on the clip that serves as the reference within each first clip group.

[0108] The grouping unit (540) uses the indexing information of each clip to determine whether there are clips with contextual similarity for each clip. Here, contextual similarity may mean that multiple clips are a single semantic group. In other words, contextual similarity means that multiple clips have a causal relationship or correlation, even if they may or may not have the same meaning.

[0109] The grouping unit (540) determines whether there are clips with contextual similarity among the clips extracted by the clip extraction unit (520) and groups them into a first clip group. The grouping unit (540) cannot determine contextual similarity among all clips in the video, but can only determine this for the clips extracted by the clip extraction unit (520). Since determining contextual similarity among all clips in the video is impossible due to token limitations, it is difficult to analyze or group the video as a whole. Therefore, the grouping unit (540) determines whether there are clips with contextual similarity among the clips extracted by the clip extraction unit (520). The grouping unit (540) uses the clips extracted by the clip extraction unit (520) as reference clips and compares the contextual similarity with clips at the time points before and after the corresponding clip. The grouping unit (540) can compare the contextual similarity of a reference clip to clips preceding other reference clips at the time points before and after the reference clip.

[0110] Here, contextual similarity can be determined through the following process. First, contextual similarity can mean semantic coherence or narrative coherence. The grouping unit (540) can use OCR (text), Voice (conversation or voice), and Context (context, context) information to determine semantic coherence or narrative coherence by whether the clips come together to create a single story or event beyond the individual meaning of each clip. For example, if the dialogue in a specific clip explains the situation in another clip, or if the events follow a structure of introduction, development, turn, and conclusion over time, the grouping unit (540) can determine that these clips have high narrative coherence and classify them into the same group.

[0111] Alternatively, contextual similarity may refer to audiovisual consistency. The grouping unit (540) determines whether multiple clips share the same spatiotemporal background using visual elements (Vision) and auditory elements (Sound) other than dialogue. Specifically, if vector information of visual elements such as characters, background, lighting, or color, or auditory elements such as background music and scene sounds across multiple clips shows a high degree of similarity, or if caption information describes consistent content, the grouping unit (540) determines that these clips have a high degree of audiovisual consistency and can classify them into the same group.

[0112] Alternatively, contextual similarity may imply event and sequential causality. The grouping unit (540) sequentially analyzes each indexing information to determine whether a clear cause-and-effect relationship, i.e., a causal relationship, exists between clips. If a clip indicating the occurrence (cause) of a specific event is followed by a clip showing a predictable outcome due to the event, the grouping unit (540) may determine that these clips have a causal relationship and classify them into the same group. The grouping unit (540) can go beyond simply grouping visually similar scenes and logically reconstruct the unfolding process of a specific event.

[0113] The grouping unit (540) determines whether there are clips with contextual similarity for each clip based on the aforementioned criteria, and groups the clips with contextual similarity into the first clip group.

[0114] Furthermore, the grouping unit (540) can determine the similarity between the reference clips within the first clip group, and further group the first clip groups (lower group concept) having similar contents into second clip groups (upper group concept). Similarly, comparing and analyzing the similarity between the first clip groups as a whole causes a considerable processing load on the artificial intelligence learning model, and is practically an operation close to impossible. Therefore, the grouping unit (540) determines the similarity between the first clip groups based on the reference clips within each first clip group. The grouping unit (540) can determine the similarity between clips based on the indexed information of each reference clip, and the similarity can be determined by one or a combination of two or more of the following.

[0115] First, the grouping unit (540) can determine the similarity between reference clips based on the similarity of vectors. The grouping unit (540) determines similarity by calculating the distance (e.g., Euclidean distance) or cosine similarity between vector information representing the visual or auditory characteristics of each clip. This is effective in grouping clips with similar overall moods or styles.

[0116] The grouping unit (540) can determine the similarity between reference clips based on keyword consistency. The grouping unit (540) analyzes the frequency and consistency of keywords contained in the captions or OCR text information extracted from each clip to determine similarity. This is useful for grouping clips that share a specific person, object, or topic.

[0117] The grouping unit (540) can determine the similarity between reference clips based on contextual similarity. Based on contextual indexing information, the grouping unit (540) can group clips belonging to a specific event or storyline (e.g., "scoring scene" or "conflict scene"). This enables grouping based on the video narrative structure, beyond the content of individual clips.

[0118] Alternatively, the grouping unit (540) may determine similarity based on conditions for grouping into the second clip group (received by the communication unit (510)). The grouping unit (540) may group each clip based on the conditions.

[0119] Accordingly, the grouping unit (540) can operate as illustrated in FIGS. 8 to 10.

[0120] FIG. 8 is a drawing illustrating an example of grouping clips by a grouping unit according to one embodiment of the present invention, and FIGS. 9 and 10 are drawings illustrating a process of generating an image by a grouping unit according to one embodiment of the present invention.

[0121] Referring to FIGS. 8 to 10, assuming that the video to be indexed is a baseball play video, the clip extraction unit (520) can extract clips of a batter hitting the ball, a clip of a batter not hitting the ball, a clip of a defender catching the ball, and a clip of a defender not catching the ball from the video.

[0122] At this time, the grouping unit (540) groups clips within a section having contextual similarity based on the reference clip into a first clip group using the indexing information indexed by the indexing unit (130). For example, a clip in a video of a batter hitting a ball may be used as a reference clip, and clips of a pitcher throwing a ball, a batter hitting a ball, an infielder or outfielder catching a ball, a batter stepping on first base, etc. may be grouped into the first clip group. The same applies to the rest. The grouping unit (540) groups clips within a section having contextual similarity into the first clip group together with the reference clip.

[0123] Furthermore, the grouping unit (540) may group the reference clips within each first clip group into additional upper-level second clip groups using indexing information. For example, the grouping unit (540) may group the reference clips of the batter hitting the ball into a second clip group of the batter hitting a home run, etc., and may group the clips of the pitcher throwing the ball into a second clip group of the specific pitcher striking out, etc. Alternatively, the grouping unit (540) may group the first clip groups into a second clip group based on a separate condition for grouping into the second clip group (received by the communication unit (510). The grouping unit (540) may group the second clip groups into multiple upper-level groups as needed.

[0124] In this way, since the first clip group or the second clip group grouped by the grouping unit (540) are all indexed by the indexing unit (530), the entire clip group can be searched or analyzed. Accordingly, the grouping unit (540) can improve the accuracy of analysis or search when performing analysis or search using the indexed video.

[0125] Referring again to FIG. 5, the memory (550) stores data received by the communication unit (510) and all data generated or generated while each component (520 to 540) operates.

[0126] FIG. 11 is a flowchart illustrating a method for a video content production device according to an embodiment of the present invention to produce video content using a RAG-based generative AI model, FIG. 12 is an exemplary diagram illustrating a method for a video content production device according to an embodiment of the present invention to produce video content using a RAG-based generative AI model, and FIG. 13 is an exemplary diagram illustrating a clip video recommendation process by a RAG system in a video content production device according to an embodiment of the present invention.

[0127] The video content production device (10) receives raw video data and a user query containing a user request (S10). As illustrated in FIGS. 11 and 12 , the raw video data may be a video of making egg bread, and the user query may be "Summarize how to make egg bread. Focus on the cooking process, not the eating process."

[0128] A video content production device (10) extracts raw video data by distinguishing each clip from a plurality of clip images, performs indexing on each extracted clip, and stores the video analysis results including each clip image, indexed information, thumbnail image, and enhanced summary data from each clip image in a video analysis / storage module (130) (S20).

[0129] In addition, the video content production device (10) uses a multi-agent system to decompose a user query into one or more sub-queries through semantic analysis based on natural language processing, and executes one or more agents to execute user requests for each decomposed sub-query (S30). At this time, the user query may be decomposed into one or more sub-queries to match the indexed information of each clip. The sub-queries may represent sub-tasks to be performed by the multi-agent system, and the multi-agent system generates a work schedule to resolve the user request, and each AI agent is executed independently or cooperatively based on the generated work schedule.

[0130] The video content production device (10) can perform parsing to automatically decompose a user query, which analyzes the user query according to grammar rules to determine the structure of a sentence, semantic analysis to determine the meaning of each word and phrase based on the parsing results and interpret the meaning of the entire query, named entity recognition to identify and classify specific entities (place, time, person, etc.) within the query, intent recognition to determine the user's intent, and relationship extraction to determine the relationship between entities within the user query.

[0131] The video content production device (10) is a multi-agent system that calls the RAG system for each sub-task, searches for clip videos in the video analysis results corresponding to the user request for each sub-query, and generates response data including a video editing code through a pre-trained artificial intelligence model based on the searched clip videos and the user query (S40). At this time, the video content production device (10) can analyze the query or sub-query input by the user to determine what characteristics it has when searching for clip videos corresponding to the user request for each sub-query, and can derive a result by reflecting the determination result. The video content production device (10) divides the characteristics of the query into first and second characteristics, and determines how much of the first characteristic and the second characteristic the queries have. The video content production device (10) can perform a search by considering both the first characteristic and the second characteristic, and can assign a weight to the search result according to each characteristic value.

[0132] Here, the AI ​​model may include a large-scale language model (LLM) or a multi-modal large-scale language model. A multi-modal large-scale language model (LLM) may refer to a large-scale language model that can understand and process relationships between different data formats, such as natural language text data, image data, audio data, and video data. A multi-modal language model may include multiple encoders that encode input data corresponding to each data format. The multi-modal language model can be trained to calculate the similarity between the encoded embedding vectors from the encoders of each data format through training data that includes data of different data formats, and to calculate higher similarity for identical pairs and lower similarity for different pairs.

[0133] The video content production device (10) post-processes response data to produce video content that meets the user's request (S50). Even when the user's query is abstract, ambiguous, or poorly structured, the video content production device (10) can produce video content that reflects the user's intent through semantic analysis based on natural language processing. The multi-agent system determines a set of sub-queries to be generated for a user query and the tools to be executed for each sub-query, and can produce new video content that meets the user's request through a RAG-based AI agent.

[0134] The video content production device (10) can automatically generate editing order information in JSON or XML format that can be used in an actual video editing program based on a list of clip images (Nodes) selected by the RAG system. Here, a node represents an individual task or process and can define the main steps or states of an agent.

[0135] The video content production device (10) can generate an editing script including the start / end time, order, transition effect, and subtitle content of each clip video, and can automatically execute a video editing task through the generated script to produce video content.

[0136] At this time, the video content production device (10) can recommend N clip videos for video production corresponding to the user query, as illustrated in FIG. 13, and can produce new video content using clip videos selected by the user.

[0137] At this time, the video content production device (10) can configure a dataset with (user query, selected clip video) as positive information and (user query, unselected clip video) as negative information, and can retrain or fine-tune the generative AI model by using the dataset configured in this way to reduce the frequency of outputting false facts or negative information.

[0138] The video content production device (10) can continuously improve the performance of the correlation between the user query of the generative AI model and the selected result (clip video) by using the user feedback information, and can secure the user's preferred data, i.e., the selected clip video, thereby enabling the production of user-customized video content.

[0139] In this way, the RAG-based generative AI model can overcome limitations such as the difficulty of reflecting the latest information in the field of large-scale language models (LLM) and errors in factual relationships (hallucinations), and in the text generation process, it can search the database where the video analysis results, which are external data of the LLM, are stored, and integrate the searched contents with the inference results of the pre-trained artificial intelligence model to generate more accurate and rich video content, thereby complementing the shortcomings of the existing LLM and suggesting new possibilities for creative activities.

[0140] FIG. 14 is a flowchart illustrating a method for a video analysis / storage module to index a video according to one embodiment of the present invention.

[0141] The clip extraction unit (520) extracts one or more clips and / or scripts from the input video (S60).

[0142] The indexing unit (530) indexes preset components of each extracted clip (S70). The indexing unit (530) indexes OCR (text), Vision (visual elements), Voice (conversation or voice), Sound (auditory elements other than conversation), and Context (context, context) information for each clip extracted by the clip extraction unit (520).

[0143] The grouping unit (540) groups clips with similar content based on the indexed information (S80).

[0144] The indexing unit (530) additionally indexes each clip group grouped by the grouping unit (540) (S90).

[0145] Although FIGS. 11 and 14 describe each process as being executed sequentially, this is merely an illustrative description of the technical idea of ​​one embodiment of the present invention. In other words, a person skilled in the art to which one embodiment of the present invention pertains may modify and apply various modifications and variations, such as changing the order described in each drawing and executing it without departing from the essential characteristics of one embodiment of the present invention, or executing one or more of each process in parallel. Therefore, FIGS. 11 and 14 are not limited to a chronological order.

[0146] Meanwhile, the processes illustrated in FIGS. 11 and 14 can be implemented as computer-readable codes on a computer-readable recording medium. A computer-readable recording medium includes all types of recording devices that store data that can be read by a computer system. That is, a computer-readable recording medium includes storage media such as magnetic storage media (e.g., ROM, floppy disks, hard disks, etc.) and optical reading media (e.g., CD-ROMs, DVDs, etc.). In addition, a computer-readable recording medium can be distributed across network-connected computer systems, so that the computer-readable codes can be stored and executed in a distributed manner.

[0147] The above description is merely an example of the technical idea of ​​the present embodiment, and those skilled in the art will appreciate that various modifications and variations can be made without departing from the essential characteristics of the present embodiment. Therefore, the present embodiments are not intended to limit the technical idea of ​​the present embodiment, but rather to explain it, and the scope of the technical idea of ​​the present embodiment is not limited by these embodiments. The scope of protection of the present embodiment should be interpreted by the claims below, and all technical ideas within a scope equivalent thereto should be interpreted as being included in the scope of rights of the present embodiment.

[0148]

[0149] CROSS-REFERENCE TO RELATED APPLICATION

[0150]

[0151] *This patent application claims priority under 35 USC § 119(a) to Korean patent applications Nos. 10-2024-0109107, 10-2024-0109117, and 10-2024-0109128, filed in Korea on August 14, 2024, all of which are incorporated by reference herein. Furthermore, this patent application claims priority in countries other than the United States for the same reasons, all of which are incorporated by reference herein.

Claims

1. In the video content production device, A user interface module that receives raw video content and user queries from a user terminal and provides a response corresponding to the received user query to the user terminal; A video analysis and storage module that divides raw video data into multiple clip videos, analyzes them, and stores the analysis results of each video; An agent execution module that analyzes a user query, decomposes it into one or more sub-queries, plans and organizes tasks for resolving the user request into sub-tasks, and executes the sub-queries to search for clips; and A RAG execution module that searches for scenes corresponding to subqueries generated by the agent execution module from the video analysis and storage module, processes the searched information, and inputs it into a generative AI model to generate final response data. A video content production device characterized by including:

2. In paragraph 1, The above RAG execution module, A search module that searches for a scene corresponding to a subquery generated by the agent execution module from the video analysis and storage module; An augmentation module that processes the information searched by the above search module into an augmented context and inputs it into a generative AI model; and A video content production device characterized by including a generation module that generates final response data using augmented context.

3. In paragraph 2, The above search module, A video content production device characterized by analyzing a query or subquery entered by a user and determining what characteristics it has.

4. In paragraph 3, The above search module, A video content production device characterized in that the query characteristics are divided into first characteristics and second characteristics.

5. In paragraph 4, The first characteristic is, A video content production device characterized by indicating a relatively comprehensive and vague degree, such as the nature, characteristics, or aspects of a specific object.

6. In paragraph 4, The second characteristic is, A video content production device characterized by indicating a relatively specific and clear degree, such as a phenomenon occurring in a specific object, the internal composition, structure, or components of a specific object.

7. In paragraph 5 or 6, The above search module, A video content production device characterized in that it analyzes a query or subquery entered by a user and determines to what extent the queries have each of the first characteristic and the second characteristic.

8. In paragraph 7, The above search module, A video content production device characterized by assigning weights to search results according to each characteristic value.

9. In paragraph 8, The above search module, A video content production device characterized in that it provides search results in order of search results that best match the query based on weighted results.

10. In a method for producing video content by a video content production device, A receiving process for receiving a user query containing raw video data and user requests; A storage process of dividing raw video data into multiple clip videos, analyzing them, and storing the video analysis results, including each clip video, a thumbnail image representing each clip video, and enhanced summary data from each clip video, in a video analysis / storage module; An execution process in which a user query is decomposed into one or more sub-queries through semantic analysis based on natural language processing using a multi-agent system, and one or more agents are executed to execute user requests for each decomposed sub-query; and A generation process that calls the RAG system for each sub-task, searches for clips in the video analysis results corresponding to the user request for each sub-query, and generates response data containing video editing code using a pre-trained artificial intelligence model based on the searched clips and the user query. A method for producing video content, characterized by including:

Images