Method for Producing Video Content using a RAG-based Generative AI Model and Computing Apparatus executing the same, Computer Program

Abstract

The present invention discloses a method for producing video content using a RAG-based generative AI model, a computing device for executing the same, and a computer program. According to one aspect of the present embodiment, a method for producing video content using a RAG-based generative AI model, performed by a computing device comprising at least one processor, is provided, comprising: an agent execution process in which, when a user query including raw video data and user requests is received, the user query is decomposed into one or more subqueries through semantic analysis based on natural language processing using a multi-agent system and a RAG system, and one or more agents are executed to execute user requests for each decomposed subquery; a RAG execution process in which clip videos corresponding to user requests for each subquery are selected based on the result of analyzing the raw video data in clip units, and response data including video editing code is generated through a pre-trained generative AI model based on the selected clip videos and user queries; and a video production process in which the response data is post-processed to produce video content that meets user requests.

Description

Technology Field

[0001] The present embodiment relates to a method for producing video content using a RAG-based generative AI model capable of producing video content requested by a user by reflecting user intent, and a computing device for executing the same. Background Technology

[0003] The content described in this section merely provides background information regarding the present embodiment and does not constitute prior art.

[0004] The field of artificial intelligence (AI) is entering a new golden age, driven by the advancement of deep learning technology and the rapid growth of research on large-scale language models over the past few years. In particular, Generative AI models, which have significantly advanced the ability to process and generate natural language, are gaining attention as they expand into various applications such as conversational chatbots, automatic summarization systems, and creative text generation tools.

[0005] Generative AI is defined as a technology that aims for the ability for machines to directly generate new data, rather than merely analyzing input text or images. Although this concept has existed for some time, it has recently garnered rapid attention thanks to advancements in deep learning and Large-Scale Language Models (LLM). Based on LLM, Generative AI is a form of artificial intelligence capable of generating new content such as text, images, audio, and video, and it is being applied in various fields ranging from everyday conversation to finance, healthcare, education, and entertainment.

[0006] In this context, the core technology of LLM is the Transformer architecture. Unlike existing language models that processed sentences sequentially, the Transformer processes the entire sentence at once, enabling it to learn the relationships between words more effectively. Through this, generative AI can move beyond simple sentence generation to perform various language-related tasks, such as complex question answering, text analysis, creative writing, translation, and summarization.

[0007] While Large Multimodal Models (LMMs) or LLMs can process various forms of data such as text, images, voice, and video, they are limited to static data and process not only language-related information but also non-verbal information like images, sound, and video. Existing LLMs or LMMs have limitations in responding to environments that change in real time or directly interacting with the physical world. This means it is difficult to analyze the current situation and decide on immediate actions accordingly. Furthermore, LLM-based generative AI processes only the information necessary for individual tasks, limiting its comprehensive understanding and prediction of the overall environment. Additionally, due to structural constraints that make it difficult to immediately reflect the latest information, it may generate answers that conflict with current data (a problem of lack of fact-based information). Moreover, because it relies on previously learned content, it may exhibit a low level of understanding regarding new problems or domains.

[0008] As a solution to this, various methods are being explored, such as utilizing domain-specific fine-tuning LLMs and RAGs to enhance reliability using customized datasets for each company. Currently, many companies are adopting AI systems in various fields, including information retrieval, content summarization, and production, by leveraging LLM and RAG technologies. However, these systems have limitations, such as data bias, hallucination issues, difficulties in real-time data processing, and a lack of knowledge in specific fields. In particular, hallucinations occur when the system fails to find an answer to a user's question; instead of providing a clear response like "I'm sorry" or "I don't know," it presents irrelevant information as if it were factual. Furthermore, there are instances where accurate answers exist but are omitted from the response list due to low search result rankings.

[0009] Existing RAG models load knowledge all at once and generate answers without rework, leading to issues with reduced accuracy and the inability to view real-time data after the model's configuration. Since knowledge information is vectorized only once and undergoes a single answer generation process, inaccurate responses may be provided without reworking the existing information, potentially resulting in lower accuracy. This problem is particularly likely to occur when generating answers to complex questions, and existing RAG models have struggled to effectively handle diverse question types. The problem to be solved

[0011] One objective of the present invention is to provide a method for producing video content using a RAG-based generative AI model capable of decomposing a user query into one or more subqueries to identify user intent using a multi-agent system and a RAG system, searching raw video data input by the user for each subquery in clip units, and providing response data that enables the production of video content corresponding to the user query through a large-scale language model based on the searched video clips and user queries, and a computing device for executing the same. means of solving the problem

[0013] According to one aspect of the present embodiment, a method for producing video content using a RAG-based generative AI model, performed by a computing device comprising at least one processor, comprises: an agent execution process in which, upon receiving a user query including raw video data and user requests, the user query is decomposed into one or more subqueries through semantic analysis based on natural language processing using a multi-agent system and a RAG system, and one or more agents are executed to execute user requests for each decomposed subquery; a RAG execution process in which clip videos corresponding to user requests for each subquery are selected based on the result of analyzing the raw video data in clip units, and response data including video editing code is generated through a pre-trained generative AI model based on the selected clip videos and user queries; and a video production process in which the response data is post-processed to produce video content that meets user requests.

[0014] Alternatively, the video analysis process further comprises the step of distinguishing and extracting each clip from the raw video data, performing indexing on each extracted clip using text (OCR), visual elements (Vision), dialogue or voice information (Voice), auditory elements other than dialogue (Sound), and context information, and then storing the indexed information of each clip. In this case, the agent execution process is characterized by decomposing the user query into one or more subqueries to match the indexed information of each clip.

[0015] Alternatively, the agent execution process further comprises a video analysis process that, after receiving the raw video data and a user query including user requests, divides the raw video data into a plurality of clip videos for analysis, and stores a video analysis result including each clip video, a thumbnail image representing each clip video, and summary data augmented in each clip video.

[0016] Alternatively, the agent execution process is characterized by generating a task schedule based on the subquery and allowing at least one agent to execute independently or cooperatively according to the generated task schedule.

[0017] Alternatively, the generative AI model is characterized as being a large-scale language model (LLM) or a multi-modal large-scale language model.

[0018] Alternatively, the RAG execution process is characterized by providing list information of clip images selected by the RAG system and providing editing order information available for use in a video editing program based on the list information.

[0019] Alternatively, the video production process is characterized by generating an editing script including the start and end times, sequence, transition effects, and subtitle content of each video clip based on the list information and editing order information, and automatically executing video editing operations through the generated script to produce video content.

[0020] Alternatively, the method further includes a dataset construction process in which, after the above video production process is performed, positive information consisting of pairs of (user query, selected clip video) is generated using clip videos used for video content production among the clip videos selected by the RAG system, and negative information consisting of pairs of (user query, unselected clip video) is generated using clip videos not used for video content production to construct a dataset.

[0021] Alternatively, it is characterized by further including a retraining process that retrains a generative AI model using the above dataset as training data.

[0022] According to one aspect of the present embodiment, a computing device for performing a method of executing an active demand response based on artificial intelligence comprises: a network unit that communicates with an external device; a memory in which at least one process for performing an operation of executing an active demand response based on artificial intelligence is stored; and a processor that performs an operation of executing an active demand response based on artificial intelligence according to the process. The processor is characterized by, when a user query including raw video data and user requests is received, using a multi-agent system and a RAG system, decomposing the user query into one or more subqueries through semantic analysis based on natural language processing, executing one or more agents for executing user requests for each decomposed subquery, selecting video clips corresponding to user requests for each subquery based on the result of analyzing the raw video data in clip units, generating response data including video editing code through a pre-trained artificial intelligence model based on the selected video clips and user queries, and post-processing the response data to produce video content that meets user requests.

[0023] Alternatively, the processor is characterized by receiving the raw video data and a user query including user requests, dividing the raw video data into a plurality of clip videos for analysis, and storing a video analysis result including each clip video, a thumbnail image representing each clip video, and summary data augmented in each clip video.

[0024] Alternatively, as a computer program stored on a computer-readable storage medium, said computer program performs operations for executing an active demand response based on artificial intelligence when executed on one or more processors, said operations are characterized by including: an agent execution operation in which, upon receiving a user query including raw video data and user requests, one or more agents are executed to execute user requests for each decomposed subquery, using a multi-agent system and a RAG system to decompose the user query into one or more subqueries through semantic analysis based on natural language processing; a RAG execution operation in which clip videos corresponding to user requests for each subquery are selected based on the results of analyzing the raw video data in clip units, and response data including video editing code is generated through a pre-trained artificial intelligence model based on the selected clip videos and user queries; and a video production operation in which video content suitable for user requests is produced by post-processing the response data. Effects of the invention

[0026] As described above, according to one aspect of the present embodiment, when raw video data and a user query are received from a user, a multi-agent system and a RAG system are used to search for video analysis results of the raw video data stored in a database, and response data is generated through a large-scale language model based on the search results, thereby generating response data that reflects the user's intent, which can provide the effect of improving the accuracy and reliability of the response data.

[0027] In addition, the present invention enables the production of more accurate and richer video content by post-processing RAG-based response data, and can provide the effect of overcoming limitations such as the difficulty of reflecting the latest information and errors regarding facts (hallucinations) by supplementing the shortcomings of existing LLMs. Brief explanation of the drawing

[0029] FIG. 1 is a block diagram of a computing device according to one embodiment of the present invention. FIG. 2 is a diagram illustrating the configuration of a computing device for executing a method of producing video content using a RAG-based generative AI model according to an embodiment of the present invention. FIG. 3 is an exemplary diagram for implementing a multi-agent system according to one embodiment of the present invention. FIG. 4 is an exemplary diagram illustrating an agent execution module and a RAG execution module according to an embodiment of the present invention. FIG. 5 is a diagram that explains in detail the configuration of a video analysis and storage module according to one embodiment of the present invention. FIG. 6 is a diagram illustrating information indexed by an indexing unit according to an embodiment of the present invention. FIG. 7 is a flowchart illustrating a method for producing video content using a RAG-based generative AI model according to an embodiment of the present invention. FIG. 8 is an example diagram illustrating a method for producing video content using a RAG-based generative AI model according to an embodiment of the present invention. FIG. 9 is an exemplary diagram illustrating a clip image recommendation process by a RAG system according to an embodiment of the present invention. Specific details for implementing the invention

[0030] The present invention is susceptible to various modifications and may have various embodiments, and specific embodiments are illustrated in the drawings and described in detail. However, this is not intended to limit the invention to specific embodiments, and it should be understood that the invention includes all modifications, equivalents, and substitutions that fall within the spirit and scope of the invention. Similar reference numerals have been used for similar components in the description of each drawing.

[0031] Terms such as first, second, A, B, etc., may be used to describe various components, but said components should not be limited by said terms. These terms are used solely for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be named the second component, and similarly, the second component may be named the first component. The term "and / or" includes a combination of a plurality of related described items or any of a plurality of related described items.

[0032] When it is stated that one component is "connected" or "connected" to another component, it should be understood that while it may be directly connected or connected to that other component, there may also be other components in between. On the other hand, when it is stated that one component is "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between.

[0033] The terms used in this application are used merely to describe specific embodiments and are not intended to limit the invention. The singular expression includes the plural expression unless the context clearly indicates otherwise. In this application, terms such as "comprising" or "having" should be understood as not precluding the existence or addition of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification.

[0034] The term “acquisition” as used in this disclosure can be understood to mean not only receiving data through a wired or wireless communication network with an external device or system, but also generating data in an on-device form.

[0035] Meanwhile, the terms "module" or "unit" used in this disclosure may be understood as referring to an independent functional unit that processes computing resources, such as a computer-related entity, firmware, software or a part thereof, hardware or a part thereof, or a combination of software and hardware. In this case, "module" or "unit" may be a unit composed of a single element, or a unit expressed as a combination or set of multiple elements. For example, in a narrow sense, "module" or "unit" may refer to a hardware element of a computing device or a set thereof, an application program that performs a specific function of software, a procedure implemented through software execution, or a set of instructions for program execution. Furthermore, in a broad sense, "module" or "unit" may refer to the computing device itself that constitutes the system, or an application executed on the computing device. However, since the above-described concept is merely an example, the concepts of "module" or "part" may be defined in various ways within the scope understandable to those skilled in the art based on the contents of this disclosure.

[0036] As used in this disclosure, the term “model” may be understood as a system implemented using mathematical concepts and language to solve a specific problem, a set of software units to solve a specific problem, or an abstract model regarding a process to solve a specific problem. For example, a neural network “model” may refer to the entire system implemented as a neural network that has problem-solving capabilities through learning.

[0037] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by those skilled in the art to which this invention pertains.

[0038] Terms such as those defined in commonly used dictionaries should be interpreted as having meanings consistent with their meanings in the context of the relevant technology, and should not be interpreted in an ideal or overly formal sense unless explicitly defined in this application.

[0039] In addition, each component, process, procedure, or method included in each embodiment of the present invention may be shared within a scope that is not technically contradictory to one another.

[0041] FIG. 1 is a block diagram of a computing device according to one embodiment of the present invention.

[0042] A computing device (10) according to one embodiment of the present disclosure is a system (10) that produces video content using a RAG-based generative AI model, and may include a server of any type and / or a device of any type.

[0043] Specifically, the computing device (10) may be a hardware device or part of a hardware device that performs comprehensive processing and computation of data, or it may be a software-based computing environment connected to a communication network. For example, the computing device (10) may be a server that performs intensive data processing functions and is an entity that shares resources, or it may be a client that shares resources through interaction with the server. Additionally, the computing device (10) may be a cloud system in which multiple servers and clients interact to comprehensively process data. Since the description above is merely one example regarding the type of computing device (100), the type of computing device (10) may be configured in various ways within a range understandable to those skilled in the art based on the contents of this disclosure.

[0044] Referring to FIG. 1, a computing device (10) according to one embodiment of the present disclosure may include a processor (11), a memory (12), and a network unit (13). However, since FIG. 1 is merely an example, the computing device (10) may include other configurations for implementing a computing environment. Additionally, only some of the disclosed configurations may be included in the computing device (10).

[0045] A processor (11) according to one embodiment of the present disclosure may be understood as a constituent unit comprising hardware and / or software for performing computing operations. For example, the processor (11) may read a computer program and perform data processing for machine learning. The processor (11) may process computational processes such as processing input data for machine learning, extracting features for machine learning, and calculating errors based on backpropagation. A processor (11) for performing such data processing may include a central processing unit (CPU), a general purpose graphics processing unit (GPU), a tensor processing unit (TPU), an application specific integrated circuit (ASICc), or a field programmable gate array (FPGA). Since the above-described types of processors (11) are merely examples, the types of processors (11) may be configured in various ways within a range understandable to those skilled in the art based on the contents of the present disclosure.

[0046] The processor (11) can train one or more artificial intelligence models using an architecture structured with layers of collected datasets and learning algorithms, and can fine-tune the trained artificial intelligence models by applying a specific dataset to them. Here, fine-tuning means updating the existing trained artificial intelligence models by modifying the architecture to train the existing trained artificial intelligence models according to the purpose.

[0047] Additionally, the processor (11) can evaluate the performance of each artificial intelligence model by performing a learning algorithm on the collected data set, which is divided into training data, test data, and verification data. At this time, the trained artificial intelligence model can be divided into various forms depending on the learning algorithm.

[0048] Generative AI models utilize artificial intelligence technology to automatically generate videos based on other forms of data, such as text, images, and videos. They can be used with AI video production tools in various ways, such as converting raw video data into new video content that reflects user intent. While these generative AI models include LLM and multimodal LLM, they may also include AI models that encompass image generation models, audio generation models, and other generative models.

[0049] The processor (11) implements a generative AI model in an agent-based manner for search augmentation generative (RAG) technology, utilizes LLM-based agents to automate the search and answer generation process, and combines multiple tools as needed to handle more complex tasks.

[0050] In addition to the examples described above, the types of data for training an artificial intelligence model and the output of the artificial intelligence model may be configured in various ways within a scope understandable to those skilled in the art based on the contents of this disclosure. Furthermore, at least one artificial intelligence model may be used, and multiple artificial intelligence models may be integrated into a single network, partially share a network, or be implemented as separate, independent networks.

[0051] A memory (12) according to one embodiment of the present disclosure may be understood as a unit comprising hardware and / or software for storing and managing data processed by a computing device (10). That is, the memory (12) may store data of any form generated or determined by a processor (11) and data of any form received by a network unit (13). For example, the memory (12) may include at least one type of storage medium among a flash memory type, a hard disk type, a multimedia card micro type, a card type memory, RAM (random access memory), SRAM (static random access memory), ROM (read-only memory), EEPROM (electrically erasable programmable read-only memory), PROM (programmable read-only memory), magnetic memory, a magnetic disk, and an optical disk. Additionally, the memory (12) may include a database system that controls and manages data in a predetermined system. Since the above-described type of memory (12) is merely an example, the type of memory (12) can be configured in various ways within a range understandable to those skilled in the art based on the contents of the present disclosure.

[0052] Memory (12) can structure and organize data, combinations of data, and program code executable by the processor (11) that are necessary for the processor (11) to perform operations. For example, memory (12) can store various data through the network unit (13) described later. Memory (12) can store program code that causes an artificial intelligence model to perform learning, program code that causes an artificial intelligence model to receive data input and perform inference according to the purpose of use of the computing device (10), and processed data generated as the program code is executed. In addition, memory (12) can store at least one database in a separate space or in the same space. The database may be stored within memory (12), or the database may be stored in a storage area managed by the processor (11). That is, the database may be stored in memory (12), stored in a memory different from memory (12), or stored in an external memory.

[0053] A network unit (13) according to one embodiment of the present disclosure may be understood as a constituent unit that transmits and receives data through any known form of wired or wireless communication system. For example, the network unit (13) may perform data transmission and reception using wired or wireless communication systems such as a local area network (LAN), wideband code division multiple access (WCDMA), long term evolution (LTE), wireless broadband internet (WIBRO), 5th generation mobile communication (5G), ultra-wide-band wireless communication, Zigbee, radio frequency (RF) communication, wireless LAN, wireless fidelity, near field communication (NFC), or Bluetooth. Since the communication systems described above are merely examples, wired or wireless communication systems for data transmission and reception of the network unit (13) may be applied in various ways other than those described above.

[0054] The network unit (13) can receive data necessary for the processor (11) to perform calculations through wired or wireless communication with any system or any client. Additionally, the network unit (13) can transmit data generated through the calculations of the processor (11) through wired or wireless communication with any system or any client. The network unit (13) can transmit output data of the artificial intelligence model, intermediate data derived during the calculation process of the processor (11), processed data, etc., through communication with the aforementioned database, server, or computing device.

[0055] FIG. 2 is a diagram illustrating the configuration of a computing device for executing a method of producing video content using a RAG-based generative AI model according to an embodiment of the present invention, FIG. 3 is an example diagram for implementing a multi-agent system according to an embodiment of the present invention, and FIG. 4 is an example diagram for explaining an agent execution module and a RAG execution module according to an embodiment of the present invention.

[0056] The computing device (10) can implement various functions of the processor (11), such as user interface functions, agent execution functions, RAG execution functions, and video analysis and storage functions, by modularizing them into modules. Specifically, as shown in FIG. 2, the computing device (10) includes, but is not limited to, a user interface module (110), an agent execution module (120), a video analysis and storage module (130), and a RAG execution module (140).

[0057] The user interface module (110) receives raw video content and a user query from a user terminal and provides an answer corresponding to the received user query to the user terminal. At this time, the user interface module (110) can receive the user's voice through a microphone and convert it into an input signal corresponding to the user query. In this case, the computing device (10) can convert the user's speech data into natural language query text by utilizing voice recognition technology such as STT (Speech-To-Text).

[0058] The agent execution module (120) can be implemented as a multi-agent system. A multi-agent system consists of two or more interacting AI agents, each AI agent performing tasks independently or in cooperation. An AI agent is an LLM-based entity that repeatedly plans and executes to achieve a goal, and each AI agent is given a persona to perform a specific role, which defines the agent's description and accessible tools.

[0059] This multi-agent system can be divided into a planning agent group (121) responsible for planning and processing to identify user intent, plan the sequence of actions to be executed to respond to user queries, and guide the user to information or services required for specific requests, and a function agent group (122) that performs defined functions based on the results of analyzing user queries. The multi-agent system can be designed modularly to facilitate expansion while minimizing conflicts between independent agents, and the modularly designed groups can provide flexibility to customize for various domains by integrating new agents or easily removing unnecessary agents.

[0060] The planning agent group (121) can perform a query analysis function to analyze and regenerate input user queries to identify user intent, a planning specification function to specifically plan and organize subtasks for tasks to resolve user requests, a collection and processing function to organize and collect results of completed subtasks, a task status tracking function to track and analyze the progress of subtasks, and a path setting function to determine the next step based on task status analysis to proceed with the plan to resolve user queries.

[0061] The function agent module (122) is composed of multiple agents for executing each of the following functions: generating subqueries based on the analysis content of raw video data and query analysis results; executing external APIs (such as external search engines) in addition to defined functions; video curation that performs the step of generating searched video content; generating prompts that are automatically input to a generative AI model; and executing and cleaning RAGs that search for clips using subqueries and generate formats such as JSON files, XML, and YAML for video curation.

[0062] In this way, the agent execution module (120) can resolve user requests by breaking down tasks corresponding to user requests into one or more subqueries (sub-tasks), assigning roles, collaborative execution, and integrating results.

[0063] The video analysis and storage module (130) divides raw video data into multiple clip videos for analysis and stores the video analysis results, including each clip video and a thumbnail image representing each clip video, in a database. At this time, the video analysis and storage module (130) can generate and store multiple thumbnail images for a single clip video when the clip video exceeds a preset time (e.g., about 4 to 5 seconds). Additionally, the video analysis and storage module (130) can reconstruct the raw video data into semantic clip videos after analysis and augment summary data in each clip video. The video analysis and storage module (130) can convert speech within the video data into text using various artificial intelligence models such as Maestra AI, Rask AI, and GitMind, and the converted text can be used for various purposes such as subtitle production, content summarization, and enhancement of search functions.

[0064] Accordingly, the video analysis and storage module (130) can provide augmented information including thumbnail images and summary data corresponding to each clip video as a video analysis result. In addition, since the video analysis and storage module (130) analyzes the video using thumbnail images of the clip videos, it has the advantage of being able to provide fast analysis results at a low cost.

[0065] FIG. 5 is a diagram that explains in detail the configuration of a video analysis and storage module according to one embodiment of the present invention.

[0066] As illustrated in FIG. 5, the video analysis and storage module (130) includes a communication unit (131), a clip extraction unit (132), an indexing unit (133), a grouping unit (134), and a database (135).

[0067] The video analysis and storage module (130) indexes the input raw video data to enable relatively fast and accurate processing when analyzing or searching the video content. The RAG execution module (140), implemented as an agent-based RAG system to be described later, searches for the indexed information of each clip in response to a user query to quickly find the corresponding clip video.

[0068] The video analysis and storage module (130) separates and extracts the video into one or more clips (or clip images) when performing indexing, and performs indexing on the extracted clips or additionally grouped clip groups using the extracted clips. By performing indexing on the clips or clip groups extracted from the video, the video analysis and storage module (130) can significantly reduce the amount of data to be indexed compared to the conventional method.

[0069] Accordingly, the video analysis and storage module (130) can quickly index the input video with less computational power. This can bring significant advantages to external devices that analyze or search for videos. These external devices use artificial intelligence models such as LLM and LMM when analyzing or searching for videos. However, these artificial intelligence models have a problem in that the accuracy of the inference value output decreases significantly and the time required to output the inference value increases considerably as the amount of data to be processed increases. Consequently, if the entire video is analyzed or searched, it takes a long time and the accuracy of the result decreases significantly. On the other hand, when indexing is performed through the video analysis and storage module (130), analysis or search can be performed using indexed clips or clip groups rather than the entire video, so the external device can infer results that are significantly faster and highly accurate.

[0070] Furthermore, the video analysis and storage module (130) performs indexing not only on the extracted clips but also on the clip groups, thereby enabling relatively fast and accurate processing when analyzing or searching video content. After extracting the clips, the video analysis and storage module (130) additionally groups the clips that correspond to each type. For example, if a baseball play video is input as the raw video to be indexed, the video analysis and storage module (130) can extract clips within the video such as a batter hitting the ball, a batter missing the ball, a fielder catching the ball, or a fielder failing to catch the ball. Furthermore, the video analysis and storage module (130) can use the extracted clips to group them into a group of clips hitting a single, a group of clips hitting a home run, a group of clips getting a strike, or a group of clips increasing the out count, and can use the grouped clip groups to additionally group them into a group of clips scoring a run or a group of clips striking out. Accordingly, the video analysis and storage module (130) can extract clips or groups of clips having various topics or information about the video, so that other devices can analyze or search the video content more accurately using the indexed information.

[0071] In some cases, the communication unit (131) may additionally receive grouping conditions from an external device when grouping each clip. As described below, the communication unit (131) may additionally receive conditions for grouping into a second clip group from an external device when grouping a first clip group into a second clip group.

[0072] The clip extraction unit (132) separates and extracts each clip from the input video. That is, the clip extraction unit (132) separates clips into the time before and after when a change exceeding a preset threshold in the pixel value of a frame or HSV occurs within the video, and the time before and after when a change in dialogue or sound occurs. Accordingly, the clip extraction unit (132) extracts at least one clip from the video.

[0073] The clip extraction unit (123) can separate the received video into frames. The clip extraction unit (132) determines whether a change exceeding a preset threshold has occurred in the pixel value or HSV (color, saturation, brightness) within each frame. The clip extraction unit (132) may determine the pixel value or HSV of the entire frame, or, for a more detailed and accurate determination, may divide the entire frame into multiple sections and determine the pixel value or HSV for each section. If a difference exceeding a preset threshold has occurred in the pixel value or HSV of the entire preceding and succeeding frames or for each section, the clip extraction unit (132) determines that the clip has changed between the analyzed preceding and succeeding frames. The clip extraction unit (132) separates the section from the frame following the last frame where the clip was previously determined to have changed to the last frame where the clip is currently determined to have changed as one clip, and separates the next clip under the same conditions. The clip extraction unit (132) separates the input video into one or more clips and extracts them in this manner.

[0074] Alternatively, the clip extraction unit (132) extracts clips based on changes in the edges of objects within each frame. The clip extraction unit (132) converts each frame into grayscale, removes basic noise, and then extracts the edges of objects present in each frame using various edge detection methods (e.g., Canny Edge Detector, Sobel Filter, or Laplacian Filter). The edges of the objects are extracted in the form of binary images. The clip extraction unit (132) extracts the edges of objects for each segment or each unit pixel between each frame and calculates their average values. Likewise, the clip extraction unit (132) accumulates the aforementioned average values ​​between the preceding and succeeding frames and determines whether the accumulated average value has changed above a preset threshold. Based on the aforementioned method, the clip extraction unit (132) extracts clips based on changes in the edges of objects.

[0075] Alternatively, the clip extraction unit (132) may operate in parallel with or optionally with the method described above, and extracts clips by distinguishing them based on changes in dialogue content or sounds other than dialogue within the video. The clip extraction unit (132) recognizes dialogue (Voice) or sounds other than dialogue within the video. In recognizing the dialogue (Voice) content, the clip extraction unit (132) may use methods that convert dialogue or voice into text, such as the Whisper model, Maestra model, Rask model, or GitMind. Representatively, the Whisper AI model receives voice data as input and extracts acoustic features by analyzing frequency, intensity, or temporal change amount within the voice data. Subsequently, the Whisper AI model recognizes features such as language or pronunciation from the extracted acoustic features and converts them into text. The clip extraction unit (132) recognizes changes in dialogue content using the converted text. Changes in the content of the conversation may occur when the sentence is terminated by a terminal ending such as 'da' or 'kka' (in the case of a language with terminal endings) or when the speaker's breathing is interrupted. As previously mentioned, when a sentence is terminated or the speaker's breathing is interrupted using text, the clip extraction unit (132) may consider that a change in the content of the conversation has occurred and extract the corresponding point in time by distinguishing it as a cut. In recognizing sound, the clip extraction unit (132) may use a method for recognizing sound events, such as a Sound Event Detection (SED) model. The Sound Event Detection model recognizes whether specific or various sound events occur, as well as the start and end points, within the input data. The clip extraction unit (132) recognizes the occurrence / start of various sound events within the video, and depending on the type of sound event, when a specific sound event ends, the clip extraction unit (132) may recognize that a change in sound has occurred.The clip extraction unit (132) can extract the corresponding point in time by dividing it into cuts.

[0076] The clip extraction unit (132) can extract a script for each extracted clip. The clip extraction unit (132) can convert the extracted clips (e.g., files in mp4 format, etc.) into an audio file (e.g., files in mp3 format, etc.) and extract text (script) from the audio file. The clip extraction unit (132) can use a Whisper model, etc., to extract the text.

[0077] The indexing unit (133) performs indexing on each clip extracted from the clip extraction unit (132), and further, on the clip groups grouped in the grouping unit (134) to be described later.

[0078] The indexing unit (133) performs indexing on each clip extracted from the clip extraction unit (132), as well as on clip groups grouped by the grouping unit (134) described later. The indexing unit (133) performs indexing so that an external device can find, analyze, or search each clip from the video received by the communication unit (131).

[0079] FIG. 6 is a diagram illustrating information indexed by an indexing unit according to an embodiment of the present invention.

[0080] The indexing unit (133) indexes OCR (characters), Vision (visual elements), Voice (dialogue or voice), Sound (auditory elements other than dialogue), and Context (context, context) information for each clip or group of clips to be indexed. Here, Context is information meaning context, and refers to semantic information generated by comprehensively analyzing the relationship between the clip and the clips before and after it, or the semantic association between objects, people, and actions included within the clip, going beyond individually analyzing the visual and auditory elements of a specific clip. For example, if a clip of a 'pitcher throwing a ball' and a clip of a 'batter hitting a ball' are consecutive, the context of these two clips can be defined as a 'pitching event'. Conventionally, when indexing a video, instead of extracting clips within the video and indexing them as in the video analysis and storage module (130), indexing was performed on an entire section or the entire content of the video. Accordingly, conventionally, when performing indexing on videos, it was sufficient to index only visual elements (Voice) and dialogue or voice information (Voice), and there was the inconvenience of a significantly increased amount of data to process when attempting to index information beyond that.

[0081] On the other hand, the indexing unit (133) performs indexing for each clip or group of clips using the five elements described above. As described above, since the clip extraction unit (132) extracts clips within the video, if only visual elements and dialogue or voice information are used for the extracted clips as in the conventional method, indexing may be performed significantly inaccurately. Considering this, the indexing unit (133) adds text (OCR), auditory elements other than dialogue (Sound), and context in addition to the aforementioned content. By adding text to the information to be indexed, the indexing unit (133) can more accurately extract clips from videos, etc., that are provided in text form along with dialogue or voice. By adding auditory elements other than dialogue to the information to be indexed, the indexing unit (133) can more clearly recognize scene transitions or changes in mood within the video. Furthermore, by adding context to the information to be indexed, the indexing unit (133) can improve the indexing accuracy that would be lower if performed by only visual elements.

[0082] The indexing unit (133) indexes caption information for text (OCR) and dialogue or voice, and both caption information and vector information for visual elements (Vision), auditory elements (Sound), and context. For visual elements (Vision), auditory elements (Sound), and context, which contain relatively more complex and numerous elements, the indexing unit (133) indexes vector information for the corresponding clip, etc., to index comprehensively, and indexes caption information to index specifically.

[0083] When indexing a clip, the indexing unit (133) can adjust the weight of the information to be indexed differently by considering what factors the clip extraction unit (132) extracted the clip. For example, when the clip extraction unit (132) extracts a clip based on pixel values ​​within each frame, HSV, or changes in the edges of an object, the indexing unit (133) can perform indexing with relatively more weight given to text (OCR), visual elements (Vision), or context. Conversely, when the clip extraction unit (132) extracts a clip based on changes in dialogue content or sounds other than dialogue, the indexing unit (133) can perform indexing with relatively more weight given to dialogue, voice information, or auditory elements (Sound).

[0084] Furthermore, the indexing unit (133) additionally performs indexing on the (first and / or second) clip groups grouped by the grouping unit (134) described later. When analyzing or searching for a video, video viewers or video editors may search for or analyze individual clips, but it is also common for them to search for entire clip groups containing similar content. In this case, if indexing is not performed on the clip groups themselves, viewers may experience significant inconvenience as they can only search for or analyze each individual clip. Accordingly, the indexing unit (133) additionally performs indexing on the clip groups grouped by the grouping unit (134) described later. The indexing unit (133) also performs indexing on the clip groups using elements identical to those of the clips. As the indexing unit (133) performs indexing on the clip groups, viewers can search or analyze not only the clips but also the clip groups they desire.

[0085] The grouping unit (134) groups clips within a certain range centered on a reference clip into a first clip group based on contextual similarity using indexed information of each clip. Furthermore, the grouping unit (134) can additionally group first clip groups that have similar content by determining the similarity between clips based on the reference clip within each first clip group. Additionally, during the process of grouping clips within a certain range centered on a reference clip into a first clip group, the grouping unit (134) can set a thumbnail image of the reference clip as a representative thumbnail image. Therefore, by using the thumbnail image of each clip or the representative thumbnail image of each clip group, it is possible to analyze the video or search for the clip video, thereby providing fast search results.

[0086] The grouping unit (134) uses the indexed information of each clip to determine whether there are clips that have contextual similarity for each clip. Here, contextual similarity may mean that multiple clips are a single group in terms of meaning. That is, contextual similarity means that multiple clips have the same meaning or, even if they are not the same, have a mutual causal relationship or association.

[0087] The grouping unit (134) determines whether there are clips with contextual similarity among the clips extracted from the clip extraction unit (132) and groups them into a first clip group. The grouping unit (134) does not determine contextual similarity for all clips in the video, but only for the clips extracted from the clip extraction unit (132). Since it is impossible to determine contextual similarity for all clips in the video due to token constraints, it is difficult to analyze or group the video as a whole. Accordingly, the grouping unit (134) determines whether there are clips with contextual similarity among the clips extracted from the clip extraction unit (132). The grouping unit (134) uses the clips extracted from the clip extraction unit (132) as reference clips and compares the contextual similarity with clips at points before and after the corresponding clip. The grouping unit (134) can compare contextual similarity for a reference clip up to clips prior to other reference clips at adjacent points before and after.

[0088] Here, contextual similarity can be determined through the following process. First, contextual similarity may refer to semantic coherence or narrative coherence. The grouping unit (134) can determine semantic coherence or narrative coherence by using OCR (text), Voice (dialogue or voice), and Context (context, circumstances) information to determine whether the clips come together to form a story or event, beyond the individual meaning of each clip. For example, if the dialogue content of a specific clip explains the situation of another clip, or if the events take on a beginning-middle-end structure according to the flow of time, the grouping unit (134) can determine that these clips have high narrative coherence and classify them into the same group.

[0089] Alternatively, contextual similarity may refer to audiovisual consistency. The grouping unit (134) determines whether multiple clips share the same spatiotemporal background by using information on visual elements (Vision) and auditory elements (Sound) other than dialogue. Specifically, if vector information of visual elements such as characters, backgrounds, lighting, or colors, or auditory elements such as background music or ambient sounds, shows high similarity across multiple clips, or if caption information describes consistent content, the grouping unit (134) may classify these clips into the same group by determining that they have high audiovisual consistency.

[0090] Alternatively, contextual similarity may refer to events and sequential causality. The grouping unit (134) sequentially analyzes each indexed piece of information to determine whether there is a clear cause-and-effect relationship, i.e., a causal relationship, between the clips. If a clip indicating the occurrence (cause) of a specific event is followed by a clip showing a predictable result resulting from that event, the grouping unit (134) may determine that these clips have a causal relationship and classify them into the same group. The grouping unit (134) can go beyond simply grouping visually similar scenes and logically reconstruct the progression of a specific event.

[0091] The grouping unit (134) determines whether there are clips with contextual similarity for each clip based on the aforementioned criteria, and groups the clips with contextual similarity into a first clip group.

[0092] Furthermore, the grouping unit (134) can determine the similarity between reference clips within the first clip group and additionally group the first clip groups (sub-group concept) having similar content into second clip groups (super-group concept). Likewise, comparing and analyzing the similarity between the first clip groups as a whole is practically impossible because it causes a significant processing load on the artificial intelligence learning model. Accordingly, when determining the similarity between the first clip groups, the grouping unit (134) determines the similarity based on the reference clips within each first clip group. The grouping unit (134) can determine the similarity between clips based on the indexed information of each reference clip, and the similarity can be determined by one or more combinations of the following.

[0093] First, the grouping unit (134) can determine the similarity between reference clips based on the similarity of vectors. The grouping unit (134) determines similarity by calculating the distance (e.g., Euclidean distance) or cosine similarity between vector information representing the visual or auditory characteristics of each clip. This is effective for grouping clips that have a similar overall mood or style.

[0094] The grouping unit (134) can determine the similarity between reference clips based on keyword matching. The grouping unit (134) determines similarity by analyzing the frequency of occurrence and matching of keywords included in the caption or OCR text information extracted from each clip. This is useful for grouping clips that share a specific person, object, or topic.

[0095] The grouping unit (134) can determine the similarity between reference clips based on contextual similarity. Based on context-indexed information, the grouping unit (134) can group clips belonging to a specific event or storyline (e.g., 'scoring scene', 'conflict scene') into a single group. This enables grouping based on the video narrative structure beyond the content of individual clips.

[0096] Alternatively, the grouping unit (134) may determine similarity based on conditions for grouping into a second clip group (received by the communication unit (131)). The grouping unit (134) may group each clip based on the said conditions.

[0097] The database (135) stores data received by the communication unit (131) and information that occurs or is generated as each component (131 to 1340) operates.

[0098] Referring again to FIGS. 2 to 4, the RAG execution module (140) can search for external data during the text generation process and integrate the searched information with pre-learned content to generate more accurate and richer content. The RAG execution module (140) can be implemented as an agent-based RAG system, and the RAG system can respond to various types of questions and can effectively generate answers even when there is a lack of knowledge about a specific domain.

[0099] The RAG system includes a search module (141), an augmentation module (142), and a generation module (143). The augmentation module (142) serves as a gateway for obtaining necessary information from external knowledge sources. Since the generative AI model may lose the timeliness or accuracy of knowledge if it relies solely on learned parameters, the search module (141) finds appropriate materials from a large set of documents or databases to compensate for this. The search module (141) utilizes embedding-based search (Vector Similarity Search) technology to convert natural language sentences or words into vector forms, and then measures the similarity (such as cosine similarity) between these vectors to search for related documents. These embeddings can be obtained from various models such as BERT and Sentence Transformers, and large-scale vector data can be efficiently searched through vector indexing libraries such as FAISS or Annoy.

[0100] The augmentation module (142) does not immediately feed the information retrieved through the search module (141) into the generative AI model, but rather processes it into an augmented context. The augmented context may consist of the results of summarizing, extracting, and filtering the retrieved information or a part of the retrieved information so that the generative AI model can refer to it when generating an answer.

[0101] The response quality of a generative AI model can vary significantly depending on how the context to be referenced is injected and how the query or answer requirements are set. Therefore, the RAG execution module (140) can derive an answer that more accurately reflects the actual user's intent by utilizing prompt design (prompt engineering) technology to connect the context of the query and the context as naturally as possible and explicitly providing the information that the model needs to reference.

[0102] The generation module (143) generates final response data using augmented context. Various large-scale language models (200), such as GPT-based, T5-based, and BERT-based generative models, can be utilized, and recently, there has been an increasing number of cases using fine-tuned dedicated models or the latest APIs. The generation module (143) generates an answer by combining information obtained from the search module (141) and the augmentation module (143) with knowledge learned internally, while maintaining factuality and contextual consistency as much as possible. The response data generated by the generation module (143) can be transmitted to a user terminal after undergoing a separate post-processing process or linked to other modules within the system.

[0103] As illustrated in FIG. 3, when a user query is "summarize how to make egg bread. Focus on cooking scenes rather than eating scenes," the agent execution module (120) breaks down the user request into one or more subqueries so that it can search more specifically. At this time, the agent execution module (120) can break down the user query into one or more subqueries based on the indexed information of each clip, and can quickly and easily search for clip videos corresponding to each subquery by searching the indexed information within the database (135) of the video analysis and storage module (130).

[0104] For example, a subquery can be broken down into the ingredient preparation scene (the scene of preparing necessary ingredients), the egg whisking scene (the scene of whisking and mixing eggs), and the bread soaking scene (the scene of soaking bread in the egg mixture).

[0105] In this way, the RAG execution module (140) searches for scenes corresponding to a subquery in clip video units containing augmented information including thumbnail images and summary data based on the video analysis results stored by the video analysis and storage module (130), and generates and provides response data in the form of text from the searched results.

[0106] At this time, the RAG execution module (140) can generate response data including indexed information of the clip image (e.g., id: "c14"), summary text and text vector (e.g., "dip the bread in the egg mixture with both hands"), script text and text vector (e.g., "Ah, now all the trouble is done once I soak the bread"), and image frames and image vectors.

[0107] FIG. 7 is a flowchart illustrating a method for producing video content using a RAG-based generative AI model according to an embodiment of the present invention, FIG. 8 is an example diagram illustrating a method for producing video content using a RAG-based generative AI model according to an embodiment of the present invention, and FIG. 9 is an example diagram illustrating a clip video recommendation process by a RAG system according to an embodiment of the present invention.

[0108] For the purpose of a concise and clear explanation of the process or step described below, the subject of each step is collectively referred to as the computing device, as it is a process performed by each functional part of the computing device or the invention.

[0109] Referring to FIGS. 7 and 8, a method for producing video content using a RAG-based generative AI model is as follows: a computing device (10) receives raw video data and a user query including user requests (S10). As shown in FIG. 5, the raw video data may be a video of making egg bread, and the user query may be, "Summarize how to make egg bread. Focus on the cooking scenes rather than the eating scenes."

[0110] The computing device (10) separates and extracts each clip from a plurality of clip videos from raw video data, performs indexing on each extracted clip, and stores video analysis results including each clip video, indexed information, thumbnail image, and summary data augmented in each clip video in a database (S20).

[0111] Additionally, the computing device (10) uses a multi-agent system to decompose a user query into one or more subqueries through semantic analysis based on natural language processing, and executes one or more agents to execute user requests for each decomposed subquery (S30). At this time, the user query can be decomposed into one or more subqueries to match the indexed information of each clip. The subqueries may represent sub-tasks that the multi-agent system must perform, and the multi-agent system generates a task schedule to resolve user requests, and each AI agent executes independently or cooperatively based on the generated task schedule.

[0112] A computing device (10) can perform parsing to determine the structure of a sentence by analyzing the user query according to grammar rules to automatically decompose the user query, semantic analysis to determine the meaning of each word and phrase based on the parsing results and interpret the meaning of the entire query, named entity recognition to identify and classify specific entities (places, times, people, etc.) within the query, intent recognition to determine user intent, and relationship extraction to determine the relationship between entities within the user query.

[0113] The computing device (10) calls the RAG system for each sub-task to search for clip images within the video analysis results corresponding to the user request for each sub-query, and generates response data containing code for video editing through a pre-trained artificial intelligence model based on the searched clip images and user queries (S40).

[0114] Here, the artificial intelligence model may include a large-scale language model (LLM) or a multi-modal large-scale language model. A multi-modal large-scale language model (LMM) may refer to a large-scale language model capable of understanding and processing relationships between different data formats, such as natural language text data, image data, audio data, and video data. A multi-modal language model may include multiple encoders that encode input data corresponding to each data format. A multi-modal language model may be trained using training data containing data of different data formats to calculate similarity between embedding vectors encoded by encoders of each data format, such that similarity between identical pairs is calculated to be higher, and similarity between different pairs is calculated to be lower.

[0115] The computing device (10) post-processes the response data to produce video content that meets the user's request (S50). Even if the user query is an abstract and ambiguous question or is not properly structured, the computing device (10) can produce video content that reflects the user's intent through semantic analysis based on natural language processing. The multi-agent system determines the set of subqueries to be generated for the user query and the tool that each subquery must execute, and can produce new video content corresponding to the user's request through a RAG-based AI agent.

[0116] The computing device (10) can automatically generate editing sequence information in JSON or XML format that can be used in an actual video editing program based on a list of clip images (Nodes) selected by the RAG system. Here, a node represents an individual task or process and can define a major stage or state of an agent.

[0117] The computing device (10) can generate an editing script that includes the start / end time, sequence, transition effects, and subtitle content of each clip video, and can automatically execute video editing operations through the generated script to produce video content.

[0118] At this time, the computing device (10) can recommend N clip videos for video production corresponding to the user query, as shown in FIG. 9, and can produce new video content using the clip videos selected by the user.

[0119] At this time, the computing device (10) can construct a dataset by using (user query, selected clip video) as positive information and (user query, unselected clip video) as negative information, and can use the constructed dataset to retrain or fine-tune the generative AI model in a direction that reduces the frequency of outputting false facts or negative information.

[0120] The computing device (10) can continuously improve the performance of the correlation between the user query of the generative AI model and the selected result (clip video) by using user feedback information, and can obtain data preferred by the user, i.e., the selected clip video, thereby enabling the production of user-customized video content.

[0121] In this way, RAG-based generative AI models can overcome limitations in the field of Large Language Models (LLM), such as the difficulty of reflecting the latest information and errors regarding facts (hallucinations). Furthermore, by searching a database containing video analysis results—which is external data to LLM—during the text generation process, and integrating the retrieved content with the inference results of a pre-trained AI model to generate more accurate and richer video content, they can complement the shortcomings of existing LLM and present new possibilities for creative activities.

[0123] Although FIG. 7 describes each process as being executed sequentially, this is merely an illustrative explanation of the technical concept of one embodiment of the present invention. In other words, a person skilled in the art to which one embodiment of the present invention belongs can modify and adapt it in various ways, such as changing the order described in each figure or executing one or more of the processes in parallel, without departing from the essential characteristics of one embodiment of the present invention; therefore, FIG. 7 is not limited to a chronological order.

[0124] Meanwhile, the processes illustrated in FIG. 7 can be implemented as computer-readable code on a computer-readable recording medium. A computer-readable recording medium includes all types of recording devices in which data that can be read by a computer system is stored. That is, a computer-readable recording medium includes storage media such as magnetic storage media (e.g., ROM, floppy disk, hard disk, etc.) and optical reading media (e.g., CD-ROM, DVD, etc.). In addition, computer-readable recording media can be distributed across networked computer systems, allowing computer-readable code to be stored and executed in a distributed manner.

[0125] The above description is merely an illustrative explanation of the technical concept of the present embodiment, and a person skilled in the art to which the present embodiment belongs would be able to make various modifications and variations within the scope of the essential characteristics of the present embodiment. Accordingly, the present embodiments are intended to explain, not limit, the technical concept of the present embodiment, and the scope of the technical concept of the present embodiment is not limited by these embodiments. The scope of protection of the present embodiment shall be interpreted by the claims below, and all technical concepts within an equivalent scope shall be interpreted as being included within the scope of rights of the present embodiment.

[0127] This invention is a technology developed through the Seoul Metropolitan Government Seoul Economic Promotion Agency's 2024 Creative Industry Technology Commercialization Support Project (CC240014) “Development of a video editing solution capable of cut editing and background music generation with just a click for the digitally disadvantaged.” Explanation of the symbols

[0129] 10: Computing device 11: Processor 12: Memory 13: Network Department 110: User Interface Module 120: Agent execution module 130: Video Analysis and Storage Module 140: RAG Execution Module

Claims

Claim 1 A method for producing video content using a RAG-based generative AI model, performed by a computing device comprising at least one processor, comprising: an agent execution process in which, upon receiving a user query including raw video data and user requests, the user query is decomposed into one or more subqueries through semantic analysis based on natural language processing using a multi-agent system and a RAG system, and one or more agents are executed to execute user requests for each decomposed subquery; a RAG execution process in which clip videos corresponding to user requests for each subquery are selected based on the result of analyzing the raw video data in clip units, and response data including video editing code is generated through a pre-trained generative AI model based on the selected clip videos and user queries; and a video production process in which the response data is post-processed to produce video content that meets user requests.The agent execution process further includes a video analysis process that, after receiving a user query including raw video data and user requests, divides and analyzes the raw video data into a plurality of clip videos, and stores a video analysis result including each clip video, a thumbnail image representing each clip video, and summary data augmented in each clip video; the video analysis process further includes a step of distinguishing and extracting each clip from the raw video data, performing indexing on each extracted clip using text (OCR), visual elements (Vision), dialogue or voice information (Voice), auditory elements other than dialogue (Sound), and context information, and then storing the indexed information of each clip; the agent execution process is characterized by decomposing the user query into one or more subqueries to match the indexed information of each clip; the agent execution process is characterized by generating a work schedule based on the subqueries and ensuring that at least one agent executes independently or cooperatively according to the generated work schedule; and the RAG execution process is characterized by providing list information of clip videos selected by the RAG system and the list information A method characterized by providing editing sequence information usable in a video editing program based on the list information and editing sequence information, wherein the video production process is characterized by generating an editing script including the start and end times, sequence, transition effects, and subtitle content of each clip video based on the list information and editing sequence information, and automatically executing a video editing operation through the generated script to produce video content. Claim 2 delete Claim 3 delete Claim 4 delete Claim 5 delete Claim 6 A method according to claim 1, wherein the generative AI model is characterized as being a large-scale language model (LLM) or a multi-modal large-scale language model. Claim 7 delete Claim 8 delete Claim 9 The method according to claim 1, further comprising a dataset construction process in which, after the video production process is performed, positive information consisting of pairs of (user query, selected clip video) is generated using a clip video used for video content production among the clip videos selected by the RAG system, and negative information consisting of pairs of (user query, unselected clip video) is generated using a clip video not used for video content production to construct a dataset. Claim 10 A method according to claim 9, further comprising a retraining process for retraining a generative AI model using the above dataset as training data. Claim 11 A computing device for performing a method of executing an active demand response based on artificial intelligence, comprising: a network unit that communicates with an external device; and a memory in which at least one process for performing an operation of executing an active demand response based on artificial intelligence is stored. The system includes a processor that performs an operation to execute an active demand response based on the artificial intelligence according to the above process; wherein, when a user query including raw video data and user requests is received, the processor performs an agent execution process in which, using a multi-agent system and a RAG system, the user query is decomposed into one or more subqueries through semantic analysis based on natural language processing, and one or more agents are executed to execute user requests for each decomposed subquery; wherein, based on the results of analyzing the raw video data in clip units, clip videos corresponding to user requests for each subquery are selected, and a RAG execution process is performed to generate response data including video editing code through a pre-trained artificial intelligence model based on the selected clip videos and user queries; and wherein a video production process is performed to post-process the response data to produce video content that meets user requests. The agent execution process further includes a video analysis process in which, after receiving the user query including the raw video data and user requests, the raw video data is divided into a plurality of clip videos for analysis, and a video analysis result including each clip video, a thumbnail image representing each clip video, and summary data augmented in each clip video is stored. The process involves distinguishing and extracting each clip from the above raw video data, and for each extracted clip, text (OCR), visual elements (Vision), dialogue or voice information (Voice),A computing device comprising: a step of performing indexing using auditory elements (Sound) and context information other than dialogue, and then storing the indexed information of each clip; wherein the agent execution process is characterized by decomposing the user query into one or more subqueries to match the indexed information of each clip; wherein the agent execution process is characterized by generating a work schedule based on the subqueries and ensuring that at least one agent executes independently or cooperatively according to the generated work schedule; wherein the RAG execution process is characterized by providing list information of clip video selected by the RAG system and providing editing order information usable in a video editing program based on the list information; and wherein the video production process is characterized by generating an editing script including the start and end times, order, transition effects, and subtitle content of each clip video based on the list information and editing order information, and automatically executing video editing work through the generated script to produce video content. Claim 12 delete Claim 13 A computer-readable recording medium storing a computer program for executing any one of the methods of paragraphs 1, 6, 9 and 10.

Images