Method and apparatus for generating image reflecting reference and recommending expression method

Abstract

A method for operating an automatic video generation apparatus may comprise the steps of: receiving company information; extracting company data from the company information; receiving a user query and extracting query data; extracting past behavior data of a user; and generating reference video information on the basis of the query data and the past behavior data and by using a video generation model trained using the company data as an input, and transmitting the reference video information to a user device.

Description

A method and device for generating images that reflect references and recommend expression methods.

[0001] According to various embodiments of the present disclosure, a method and device for automatically generating a video based on corporate information are disclosed. Furthermore, according to various embodiments of the present disclosure, a method and device for generating a video by incorporating user-provided references into the video generation and recommending an expression method are disclosed.

[0002] In modern society, the proliferation of smart devices is changing how we consume information. In particular, as the attention spans of companies' target consumers and employees gradually decline, preference for long-form text content and long-running video content is declining significantly. For example, preference for short-form video content with shorter playback times is increasing.

[0003] As individuals' information consumption patterns change, so too do corporate business practices. In a rapidly changing market, economy, and regulatory landscape, companies are increasingly required to produce and distribute core videos that effectively communicate and inform individuals, such as consumers and employees, to facilitate rapid communication and decision-making.

[0004] In modern society, companies possess vast amounts of corporate information. Therefore, in rapidly changing market conditions, there is a growing need for an artificial intelligence (AI) system that effectively reflects corporate information and automatically generates images that reflect the company's business characteristics.

[0005] The above information may be provided as background art to aid in understanding the present disclosure. No claim or determination is made as to whether any of the above is applicable as prior art related to the present disclosure.

[0006]

[0007] AI-based video generation technology has recently made groundbreaking progress, allowing video generation foundation models to generate videos by reflecting requests made in natural language without any specific format.

[0008] However, in practice, videos created using the Foundation model often require additional review, editing, and customization rather than being directly applied to business. This is because the Foundation model lacks the specific context or expertise of the business, or is difficult to fine-tune to meet specific business requirements. For example, when creating a commercial video for a specific company, the video must reflect the company's unique brand characteristics and incorporate the company's desired tone, manner, and visual style. These requirements cannot all be specified within the Foundation model.

[0009] Especially in corporate video, specialized messaging tailored to different business domains (e.g., insurance, finance, healthcare, IT, etc.) is required. However, creating unique content that reflects the characteristics of each business domain requires specialized knowledge and skills, and manual implementation has its limitations.

[0010] Moreover, modern businesses generate and collect diverse forms of data, which can exist in various formats, including text, images, and video. However, comprehensively processing and analyzing this diverse data is technically complex, and this can be particularly challenging when converting it into business intelligence and video content tailored to specific business domains.

[0011] Furthermore, for modern businesses, deciding which presentation method to use during the video content creation process can be a challenging task. Selecting the right presentation method can be particularly crucial for delivering a precise message tailored to business intelligence or a specific business domain.

[0012] According to various embodiments of the present disclosure, there is provided a method and device for automatically generating a corporate video based on corporate information, which supports a corporate entity to more effectively convey its expertise and uniqueness by reflecting references, recommending expression methods, and automatically generating video content optimized for corporate needs.

[0013] According to various embodiments of the present disclosure, an object is to provide a method and device for automatically generating a corporate video based on corporate information, which efficiently generates corporate-suitable video content by simultaneously analyzing and utilizing various types of data through a multimodality search function.

[0014] According to various embodiments of the present disclosure, when a user produces corporate video content based on corporate information, an optimal video expression method that matches the characteristics and goals of the company is recommended, thereby contributing to shortening the video content production time while conforming to the intended purpose of the video production, thereby providing a method and device for automatically producing corporate video content based on corporate information.

[0015] However, the problem to be solved in this disclosure is not limited to the problem mentioned above, and may be expanded in various ways without departing from the spirit and scope of this disclosure.

[0016]

[0017] According to various embodiments of the present disclosure, a method of operating an automatic video generation device may include: receiving corporate information; extracting corporate data from the corporate information; receiving a user query and extracting query data; extracting past behavioral data of the user; and generating reference video information based on the query data and the past behavioral data using a video generation model learned with the corporate data as input, and transmitting the generated reference video information to a user device.

[0018] The above reference video information may include information about a video template and information about a video content style.

[0019] The above reference video information may further include information about sample video content.

[0020] If the similarity between the query data and the past behavior data is greater than a specified threshold value, the reference video information may include reference video information included in the past behavior data.

[0021] The operating method of the above video automatic generation device may include: a step of generating a prompt based on the query data; a step of generating a context based on reference video information selected by a user; and a step of generating a video script based on the prompt, the query data, and the context.

[0022] The operating method of the automatic video generation device may further include a step of generating video content based on the video script; a step of transmitting the video content to a user device; and a step of receiving completion information or modification information input by the user.

[0023] The operating method of the automatic video generation device may further include, when receiving modification information from the user, a step of reproducing the video content by modifying at least one of object information and background information of the video content based on the modification information and transmitting the reproducing video content to the user device.

[0024] A video automatic generation device comprises at least one processor; and a memory storing instructions, wherein the instructions, when individually or collectively executed by the at least one processor, cause the processor to: receive corporate information, extract corporate data from the corporate information, receive a user query and extract query data, extract past behavioral data of the user, and generate reference video information based on the query data and the past behavioral data using a video generation model trained with the corporate data as input, and transmit the generated reference video information to a user device.

[0025] A computer program may be stored in a computer-readable recording medium to perform the operating method of the automatic video generation device according to any one of claims 1 to 7, in combination with hardware.

[0026] The automatic video generation system includes a user device that receives a user query; and an automatic video generation device, wherein the automatic video generation device extracts corporate data from the corporate information, receives a user query and extracts query data, extracts past behavioral data of the user, and uses a video generation model learned with the corporate data as input to generate reference video information based on the query data and the past behavioral data, and transmits the generated reference video information to the user device.

[0027]

[0028] According to various embodiments of the present disclosure, customized video content can be automatically generated based on a company's business intelligence data, reflecting references and aligning with the company's strategic goals. This allows the company to deliver a quick and consistent message and achieve more effective communication. Furthermore, this can enhance the company's marketing efficiency and provide video content that leaves a strong impression on target consumers.

[0029] According to various embodiments of the present disclosure, by recommending an expression method optimized for the business domain of a company, the company's expertise and reliability can be enhanced, and accordingly, the company can easily produce customized content reflecting the characteristics of the business domain and strengthen its position in the market with competitive content.

[0030] According to various embodiments of the present disclosure, by supporting a multimodality search function that can comprehensively analyze and process various forms of data such as text, images, audio, or video, a large amount of data can be efficiently utilized, and by generating video content that reflects various aspects of the data, the accuracy and comprehensiveness of information transmission can be maximized.

[0031] According to various embodiments of the present disclosure, when a user creates corporate video content, the efficiency of the content creation process can be improved by recommending presentation methods optimized for the company's business intelligence and business domain. Accordingly, the present disclosure can help users create optimal video content with less time and effort, and can enable companies to more effectively implement their marketing and communication strategies.

[0032] The effects that can be obtained from the present disclosure are not limited to the effects mentioned above, and other effects that are not mentioned can be clearly understood by a person having ordinary skill in the art to which the present disclosure belongs from the description below.

[0033]

[0034] FIG. 1 is a conceptual diagram illustrating a device for automatically generating a corporate video based on corporate information according to various embodiments of the present disclosure.

[0035] FIG. 2 is a flowchart illustrating a method for automatically generating a corporate video based on corporate information according to various embodiments of the present disclosure.

[0036] FIG. 3 is a block diagram illustrating configurations of a video generation device according to various embodiments of the present disclosure.

[0037] FIG. 4 is a block diagram showing configurations of a video generation device according to various embodiments of the present disclosure.

[0038] FIG. 5 is a flowchart illustrating a method of generating a content block and a caption for the content block in a service server according to various embodiments of the present disclosure, and extracting a candidate content block for a user query using the caption.

[0039] FIGS. 6 to 8 are flowcharts for explaining an operation method of an automatic video generation device according to various embodiments of the present disclosure.

[0040] FIG. 9 is a block diagram illustrating a user device according to various embodiments of the present disclosure.

[0041]

[0042] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. In connection with the description of the drawings, the same or similar reference numerals may be used for identical or similar components. Furthermore, in the drawings and related descriptions, descriptions of well-known functions and configurations may be omitted for clarity and conciseness.

[0043]

[0044] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another component (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0045] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0046] Various embodiments of the present document may be implemented as software (e.g., a program) including one or more instructions stored in a storage medium (e.g., memory) readable by a machine or device. For example, a processor of the machine or device may call at least one command among the one or more commands stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one command called. The one or more commands may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' only means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and this term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily in the storage medium.

[0047] According to one embodiment, the method according to various embodiments disclosed in this document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) through an application store or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0048] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0049] The processor in the present disclosure may refer to hardware capable of performing functions and operations according to each name described in the present specification, may refer to computer program code capable of performing specific functions and operations, or may refer to an electronic recording medium equipped with computer program code capable of performing specific functions and operations. According to an embodiment, the operation of the processor may be defined and / or interpreted as the operation of a video generation device, but is not limited thereto. The processor may refer to a functional and / or structural combination of hardware for performing the technical idea of ​​the present disclosure and / or software for driving the hardware.

[0050] The large language model (LLM) (hereinafter referred to as 'LLM') in the present disclosure may be a language model capable of performing natural language processing (NLP) tasks.

[0051] In this disclosure, the foundation model or basic model may be a large-scale model that generates videos using artificial intelligence technology. The foundation model may generate videos based on text descriptions, images, or other inputs.

[0052] The automatic video generation device in the present disclosure may be defined and / or referred to as a service server, which may mean one physically independent server, but is not limited thereto, and may also be one virtual machine, and may be configured to encompass one module or program or docker operating on one virtual or physical machine.

[0053]

[0054] FIG. 1 is a conceptual diagram illustrating a device for automatically generating a corporate video based on corporate information according to various embodiments of the present disclosure.

[0055] The embodiment of Fig. 1 can be combined with the embodiments of Figs. 2 to 11.

[0056] Hereinafter, a device (100) for automatically generating a corporate-related video based on corporate information may be referred to as an automatic video generation device (100).

[0057] Referring to FIG. 1, the automatic video generation device (100) may include a processor (110) and a memory (120).

[0058] Although not shown, the automatic video generation device (100) may further include a communication module for communicatively connecting to a user device (200) or a database (300) via wireless or wired communication. The communication module may include communication circuitry.

[0059] The automatic video generation device (100) may be implemented as a service server, but is not limited thereto. For example, the automatic video generation device (100) may be provided as an independent service server connected to a separate service server including a video generation model, or as an independent physical device. Hereinafter, a case in which the automatic video generation device (100) is provided as a service server including a video generation model (111) will be described as an example. In some embodiments, the automatic video generation device (100) and the user device (200) may be interpreted as being included in an automatic video generation system.

[0060] The automatic video generation device (100) can collect enterprise information from a database (300) containing enterprise information. For example, the enterprise information may include, but is not limited to, web pages, documents (e.g., PDFs, etc.), images, videos, or audio provided by users.

[0061] Corporate information may include information about the company's brand identity (BI) (hereinafter referred to as 'BI') or information about the company's business domain.

[0062] For example, a company's BI information may include, but is not limited to, at least one of the following: a logo symbolizing the company, a unique color combination associated with the brand, preset fonts and typefaces, design elements including graphics, icons, and patterns, and text data on brand personality, core values, mission and vision, positioning, brand story, and communication style. For example, a company's BI information may include information on the company's established tone and manner, information on the company's storytelling, information on the company's products and services, and data on brand messages and concepts used in marketing channels.

[0063] For example, information about a company's business domain may include, but is not limited to, information about the industry to which the company belongs. For example, information about a company's business domain may include data about the regulatory landscape (e.g., legal regulations), market trends, technical requirements, or unique characteristics of the business domain.

[0064] The automatic video generation device (100) can collect corporate information including corporate BI information or corporate business domain information from a database (300), and can collect this information at preset intervals or whenever new information (new data) is generated.

[0065] The video generation model (111) of the automatic video generation device (100) can implement an artificial intelligence-based learning model by performing learning through corporate BI data (e.g., vector data) or corporate business domain data (e.g., vector data) obtained by preprocessing collected corporate information.

[0066] The processor (110) may include a video generation model (111). The processor (110) may use the video generation model (111) to generate a business-related video based on a user query input from a user device (200).

[0067] The processor (110) may include, but is not limited to, at least one of a microprocessor, a central processing unit, a processor core, a multi-core processor, a multiprocessor, an application-specific integrated circuit (ASIC), or a field programmable gate array (FPGA), as an example.

[0068] The video generation model (111) may be implemented as part of the processor (110), but is not limited thereto. For example, the video generation model (111), or at least one instruction causing the operation of the processor (110) and / or the automatic video generation device (100), may be stored in the memory (120).

[0069] The memory (120) can store instructions (or programs) executable by the processor (110). The memory (120) can include volatile memory or non-volatile memory.

[0070] Volatile memory can be implemented as dynamic random access memory (DRAM), static random access memory (SRAM), thyristor RAM (T-RAM), zero capacitor RAM (Z-RAM), or twin transistor RAM (TTRAM).

[0071] Nonvolatile memory can be implemented as Electrically Erasable Programmable Read-Only Memory (EEPROM), flash memory, Magnetic RAM (MRAM), Spin-Transfer Torque (STT)-MRAM, Conductive Bridging RAM (CBRAM), Ferroelectric RAM (FeRAM), Phase change RAM (PRAM), Resistive RAM (RRAM), Nanotube RRAM, Polymer RAM (PoRAM), Nano Floating Gate Memory (NFGM), holographic memory, Molecular Electronic Memory Device, or Insulator Resistance Change Memory.

[0072] A user device (200) may be defined as a device used by a user (e.g., a corporate manager or corporate worker), and the user may be communicatively connected to a video generation device (200) through the user device (200). The user device (200) may include various devices such as a smart device (e.g., a smartphone), a personal computer (PC), or a portable laptop computer, which may include an application (210) (or program).

[0073] A user can input a user query (e.g., information about video creation, etc.) into a video creation device (100) through an application (210) built into a user device (200) and receive a company-related video created from the video creation device (100).

[0074]

[0075] FIG. 2 is a flowchart illustrating a method for automatically generating a corporate video based on corporate information according to various embodiments of the present disclosure.

[0076] The embodiment of FIG. 2 can be combined with the embodiment of FIG. 1, or the embodiments of FIGS. 3 to 11.

[0077] The method for automatically generating a corporate-related video (hereinafter referred to as the “automatic video generation method”) described with reference to FIG. 2 may be defined as a method performed by the video generation model (111), processor (120), or video generation device (100) of FIG. 1, but is not limited thereto.

[0078] The method for automatically generating a video may include a step (S110) of receiving corporate content.

[0079] Corporate content may include corporate information, including corporate BI information or corporate business domain information. For example, the video generation model (111) may collect corporate webpages, corporate documents (e.g., PDFs), corporate-related images, corporate-related videos, or corporate-related audio from the database (300).

[0080] Corporate content or corporate information may be defined as, but is not limited to, various data sources provided by the company. For example, corporate content or corporate information may be defined as data directly collected by the administrator of the automatic video generation device (100) with permission from the company's users. Depending on the embodiment, corporate content or corporate information may include information related to the company's users (e.g., information about similar companies) from data randomly extracted from the Internet by the automatic video generation device (100).

[0081] The method for automatically generating a video may include a step (S120) of embedding corporate content.

[0082] The step of embedding corporate content (S120) may include a step of classifying the collected corporate content or corporate information by format. For example, the embedding step (S120) may analyze the format of the collected corporate content and classify it into various multi-modalities, such as web page modality, text modality, image modality, audio modality, or video modality.

[0083] Step S120 of embedding corporate content can vectorize and process the various formats of multi-modality using the visual-language model (VLM) of the video generation model (111). That is, each corporate content can be converted into a vector representation of information contained in the classified modality. The vector representations can be defined and / or referred to as corporate embedding data that has completed embedding.

[0084] Enterprise embedding data may include vectors containing enterprise BI information (e.g., enterprise BI data) that include vectors for enterprise BI information included in the modality, or vectors containing enterprise business domain information (e.g., enterprise business domain data).

[0085] In one embodiment, the step of embedding corporate content (S120) may include a step of converting various formats of multi-modality into vector representations by vectorizing them, and a step of inserting or recording caption information into content blocks (e.g., specific portions or specific units of each content) of various formats of multi-modality.

[0086] Each modality's unit content block may include caption information for each of the various multi-modalities. Caption information may be text information, such as metadata, including, but not limited to, representative image information, title information, or summary information of the corporate content included in the content block.

[0087] The method for automatically generating a video may include a step (S130) of storing corporate embedding data in a DB.

[0088] The step (S130) of storing corporate embedding data in a database may include a step of adjusting each vector piece of information to be included in the same vector space, reflecting the characteristics of the vector-processed modality information through VLM. For example, the vector representations of different types of multi-modality, such as web pages, text, images, audio, or video, may be adjusted and integrated into a single high-dimensional vector (hereinafter referred to as an "integrated vector").

[0089] The step (S130) of storing corporate embedding data in a DB may include a step of storing data integrated into a single high-dimensional vector in a memory (130) or an external server.

[0090] The step (S130) of storing corporate embedding data in a DB may include a step of storing modality information including caption information by being caption-blocked in a memory (130) or an external server.

[0091] In the step of storing corporate embedding data in a database (S130) or in other steps, a foundation model or LLM (Large-Scale Language Model) may be trained using the corporate embedding data and trained to generate and output corporate-related videos according to a user's request. For example, the corporate embedding data may be information or data generated by performing multimodal analysis using at least one or two or more of text data, image data, audio data, and video data extracted based on corporate information.

[0092] Although not illustrated in FIG. 2, the method for automatically generating a video may include a step of extracting corporate business domain information from corporate content and generating the same into a knowledge graph.

[0093] For example, domain-specific terms can be extracted from corporate content, entities can be defined based on the extracted terms, and relationships between entities can be connected to create a knowledge graph of information specific to the business domain to which the corporate belongs.

[0094] These corporate domain-specific data and knowledge graphs for domain-specific data can be used to train LLMs that generate scenarios. More specifically, LLMs can be fine-tuned to generate scenarios specific to the financial, insurance, education, or steel industries by learning corporate domain-specific data.

[0095] The method for automatically generating a video may include a step (S140) of receiving a user query.

[0096] The step of receiving a user query may include a step of receiving query information for video generation by the user inputted into an application (210) of a user device (200). For example, the application (210) may include a video generation application. The query information is information inputted by the user and may include text information, voice information, image information, or other data information. As an example, the user may input text such as "Create a 20-second promotional video for our company's new product," and the text information may be transmitted as query information to the automatic video generation device (100).

[0097] The method for automatically generating a video may include a step (S150) of embedding a received user query.

[0098] The step of embedding a user query (S150) may include a step of converting the user query into a vector representation and a step of modality-adjusting the user query information converted into the vector representation. The user query information including the modality-adjusted vector representation may be positioned within the same vector space as the integrated vector of multiple modalities.

[0099] User query information including modality-adjusted vector representations may be defined and / or referred to as query embedding data.

[0100] The method for automatically generating a video may include a step (S160) of searching for corporate embedding data similar to query embedding data.

[0101] The step (S160) of searching for corporate embedding data similar to query embedding data may include searching for similar corporate embedding data from a memory (130) or an external server where corporate embedding data is stored. This may be performed by calculating a similarity score between the query embedding data and the corporate embedding data and extracting a similarity search result. The similarity search result may include information about the similarity scores calculated by calculating the similarity scores between the query embedding data and the corporate embedding data.

[0102] The automatic video generation method may include a step (S170) of extracting a candidate set based on the generated similarity test results.

[0103] The step (S170) of extracting a candidate set based on the similarity search result may be performed as a step of extracting, for example, at least one corporate embedding data with high relevance in order of high similarity score based on the calculated similarity test result.

[0104] In one embodiment, the step (S170) of extracting a candidate set may include selecting the corporate embedding data with the highest weighted similarity score as the final candidate set by applying a weight set by a user or administrator to a plurality of corporate embedding data with similarity scores exceeding a threshold value. The corporate embedding data selected as the final candidate set may be referenced in the video generation process and utilized to generate a corporate-related video requested by the user.

[0105] In another embodiment, the step of extracting a candidate set (S170) may include selecting a plurality of corporate embedding data having a similarity score greater than or equal to a threshold value, or a plurality of corporate embedding data having a weighted similarity score greater than or equal to the threshold value, as a final candidate set. The plurality of corporate embedding data selected as the final candidate set may be referenced in a plurality of video generation processes, thereby generating a plurality of corporate-related videos requested by the user. Accordingly, the user may select a video that aligns with the user's preference from among a plurality of videos provided by the automatic video generation device.

[0106] The method for automatically generating a video may include a step (S180) of passing a prompt, query embedding data, and context to an LLM (large-scale language model).

[0107] Prompts can be generated based on query embedding data. For example, if the query embedding data includes information such as "Create a 20-second promotional video for our company's new product," the prompt may, for example, include an instruction to the LLM to create a video script containing information about the product. The automatic video generation device (100) or the video generation model (111) may include a prompt generation module or a prompt generation circuit.

[0108] Additionally, the prompt may include information about the corporate embedding data selected for the final candidate set.

[0109] Prompts may be created by, but are not limited to, preset prompt templates.

[0110] Query embedding data and context can be passed to LLM along with prompts.

[0111] Context may include background information or contextual information related to the query embedding data. For example, context may include the context of the query embedding data itself, or information about previously provided information or related data by the user, enabling the LLM to generate a consistent response. The automatic video generation device (100) or the video generation model (111) may include a context generation module or a context generation circuit.

[0112] The method for automatically generating a video may include a step (S190) of transmitting a response generated from an LMM.

[0113] The step of transmitting the response generated in LLM (S190) may include a step of generating a video generation script in LLM based on the prompt, query embedding data, and context.

[0114] The step (S190) of transmitting the response generated in the LLM may include a step of transmitting a video generation script to a user device or a step of transmitting it to a basic model (e.g., the basic model (520) of FIG. 3).

[0115] In one embodiment, when a response generated from the LLM is passed to the base model, the output generated from the base model based on the video generation script (e.g., a company-related video requested by the user) may be transmitted to the user device (200) and / or application (210).

[0116] In another embodiment, when transmitting a response generated from LLM to a user device, information about a video generation script and information necessary for video generation (e.g., corporate embedding data, etc.) may be received from a video automatic generation device (100), and a video may be generated based on the information.

[0117]

[0118] FIG. 3 is a block diagram illustrating configurations of a video generation device according to various embodiments of the present disclosure.

[0119] The embodiment of FIG. 3 can be combined with the embodiments of FIGS. 1 to 2, or the embodiments of FIGS. 4 to 11.

[0120] Referring to FIG. 3, the automatic video generation device (100) may include a video generation model (111) (e.g., the video generation model (111) of FIG. 1).

[0121] The video generation model (111) can be communicatively and / or operatively connected to a user device (200) including an application (210) or a database (300) in which corporate information is stored.

[0122] The video generation model (111) may include a content block generation unit (410), an embedding module (420), an external content index (430), a vector DB (440), a needs prediction unit (450), a content block search unit (460), a script generation unit (470), or a CSS editing unit (480). The video generation module (111) may be communicatively and / or operatively connected to an external content repository (510), a base model (520), or an LLM (530) (e.g., a large-scale language model), but is not limited thereto. For example, the external content repository (510), the base model (520), and the LLM (530) may be included in the video generation model (111) and may be defined and / or interpreted as some components of the video generation model (111).

[0123] The operations performed by the components of the video generation model (111) can be defined as operations performed by the video generation model (111) by an instruction when at least one instruction stored in a memory (e.g., memory (130) of FIG. 1) is executed by a processor (e.g., processor (120) of FIG. 1).

[0124] The content block generation unit (410) extracts main objects from corporate information including corporate content, for example, from each frame of an image or video, divides them into semantic units, and based on the information obtained from object extraction and semantic unit division, can generate captions as metadata using a visual-language model (VLM) or an LLM that performs the function of a VLM.

[0125] According to an embodiment of the present invention, the content block generation unit (410) can divide any content and generate a plurality of content blocks. The content block generation unit can identify the logical flow of any content, the change points in the storyline, and divide the content based on the change points to generate content blocks. Alternatively, the content can be divided and generated based on the rate of change in scene change or camera angle, the change points in the background or setting, the change points in the speaker or character, or the change points in the format or style. As another example, the content block generation unit can divide the content and generate content blocks based on the point where the mood or emotional line of the video in the content significantly changes, the point where the structural elements of the video such as the intro, main text, and conclusion change, the point where the subtitle or overlay text changes, or the point where the background music or sound effect changes.

[0126] According to an embodiment of the present invention, the content block generation unit (410) can generate captions for content blocks using VLM or LLM. The content block captions can include text providing a brief description or summary of each segment or block of the video.

[0127] The content block generation unit (410) can extract a representative image of a content block or a representative image feature of a content block and control the LLM to generate a data annotation for the content block.

[0128] More specifically, the content block generation unit (410) may apply a first prompt to "generate a question about the content block and generate a response to the question in a step-by-step manner" while transmitting the content block and a representative image of the content block to the LLM trained to understand the image or video.

[0129] For example, the content block generation unit (410) may apply the above prompt to the LLM along with the cooking tutorial content block. The LLM may generate a question, "Summarize the above content block," and, to generate a response to the question, generate a first question, "What is the subject of the above content block?" and generate a first response, "It is a cooking recipe." Then, the LLM may refer to the first response to generate a second question, "What is the cooking recipe covered in the above content block?" and generate a second response, "It is carbonara pasta." Then, the LLM may refer to the second response to generate a third question, "What are the ingredients for making carbonara pasta?" and generate a third response, "It is pasta, eggs, and cheese."

[0130] A third response can be generated. Then, LLM can refer to the third response and generate a fourth question, “What is the cooking order of pasta, eggs, and cheese?” and generate a fourth response to it.

[0131] Afterwards, the content block generation unit (410) can apply a second prompt to the LLM: “Generate a caption for the content block using the generated question-answer set.”

[0132] In the previous example for the cooking tutorial content block, LLM could generate caption sentences for the content blocks such as "The topic is a cooking recipe", "The cooking recipe is about pasta carbonara", and "The ingredients for making pasta carbonara are pasta, eggs, and cheese".

[0133] The caption for the content block generated in this way is stored as metadata for the content block and is linked to the content block, and can be used for searching the content block.

[0134] Meanwhile, the content block generation unit (410) can generate a single content block by combining the generated caption and the extracted object. The content block may be a result of integrating visual content and textual descriptions (e.g., visual content generated from metadata or descriptions of its main objects). The content block may be utilized in the video generation task in the basic model (520) or may be utilized in the video content block search unit (460) to search for video content blocks.

[0135] The extracted main object may be, but is not limited to, a representative image of the video, a title of the video, a summary of the video, or a feature object of the video. The generated caption may include, but is not limited to, a caption for the representative image of the video, a caption for the title of the video, a caption for the summary of the video, or a caption for a feature object of the video. The caption may be generated using metadata of the external content index (430), but is not limited thereto.

[0136] The generated content block may be stored in an external content storage (510) along with the content or caption, but is not limited thereto.

[0137] The generated content block may be processed into a high-dimensional vector by the embedding module (420) and stored in the vector DB (440).

[0138] The embedding module (420) can analyze data of various modalities (e.g., web pages, documents, images, videos, or audio, etc.) of corporate content collected from the database (300) through an extractor of a CNN-based model, embed them, and convert them into vectors. The embedding module (420) can convert vectors of each modality into the same vector space through modality adjustment. That is, by adjusting the size or scale of the vectors of each modality, the adjusted vectors of each modality can be combined into a single integrated vector in the same vector space. Accordingly, a similarity search that comprehensively considers vectors of different modalities can be enabled. The integrated vector can be stored in a vector DB (database) (440). In addition, the foundation model (520) or LLM (530) (large-scale language model) can learn the integrated vector.

[0139] The embedding module (420) can embed the preprocessed user query and convert it into a high-dimensional vector when the user query entered by the user through the user device (200) and application (210) is preprocessed by the preprocessing module and transmitted.

[0140] The content block search unit (460) can analyze a user query vector to determine its intent. For example, if a user inputs a query such as "Create a 20-second video introducing our company's new product," the content block search unit (460) can understand the intent of the query and extract information for searching for content blocks related to the query. The content block search unit (460) can analyze the query to search for content blocks related to the type of content desired by the user. This can be performed by searching for content blocks with a vector most similar to the user's query. Based on the similarity search results, the content block search unit (460) can select and provide to the user the content blocks most suitable for the user's query. The user can check the provided content blocks and, if necessary, create a more detailed user query based on the provided content blocks and re-enter it.

[0141] The needs prediction unit (450) can extract past behavior data based on the user's past usage history. The needs prediction unit (450) can analyze the user's query vector and past behavior data to recommend personalized video templates and video content styles to the user in the form of selectable information. For example, a video template can provide a framework for how the video content to be generated will be structured. The video content style can include information about the video layout, video background color, and font of text included in the video. For example, the needs prediction unit (450) can recommend a video template or video content style preferred by the user based on the query vector and past behavior data. For example, if there is past behavior data that is substantially similar to a query included in the query vector (for example, if the similarity between the two is above a specified threshold value), information about the video template and information about the video content style included in the past behavior data can be recommended.

[0142] The needs prediction unit (450) can generate sample video content based on personalized video template information and personalized video style information. The generated sample video content can be provided to the user, and the user can check the sample video content provided by the needs prediction unit (450) and, based on the video template information and video content style information, select whether to generate a video or modify the video template or content style.

[0143] Sample video content may include video content that reflects reference information related to the company. For example, such reference information may include the company's business intelligence (BI) information or business domain information. Accordingly, the sample video content provided to users may reflect the user's company's BI or business domain information, resulting in sample video content that reflects the company's strategic goals or the specific characteristics of the company's industry, or final video content generated based on user queries. Consequently, users can receive more personalized and relevant video content, which can increase customer engagement or effectively support the company's business objectives. Consequently, relevant video content can provide advantages in corporate marketing, internal communication, and competitiveness enhancement.

[0144] The script generation unit (470) can transmit prompts, user query vectors, and context for writing a script to the LLM (530). The prompts can include information on guidelines that determine the direction and content of the script to be written by the LLM (530). The context can include information on the video template selected by the needs prediction unit (450), information on the video content style, or the company's BI information or business domain information.

[0145] The LLM (530) can generate a script based on prompts, user query vectors, and context. The script can include the basic structure and core messages of the video content. The LLM (530) is trained using data extracted from the company's BI information or business domain information. Thus, the script generated by the LLM (530) can reflect the company's BI characteristics or business domain information to generate a highly relevant script for the video content requested by the user through the user query.

[0146] The script generation unit (470) can receive a response (e.g., a script) generated through the LLM (530) and ultimately manipulate it to match the information selected by the needs prediction unit (450) and corporate data to generate a final script.

[0147] The basic model (520) can receive the final script and generate video content requested by the user based on it.

[0148] Video content generated through the basic model (520) can be provided to the user through a user device (200).

[0149] In some embodiments, the final script may be transmitted to an application (210) of the user device (200), and the user device (200) may generate video content requested by the user based on the final script.

[0150] The CSS (Cascading Style Sheets) editing unit (480) can allow users to edit generated video content using a CSS-based editor. For example, after reviewing the provided video content, the user can control the video content's project file, background rendering (e.g., background image, background color, or specific visual effects), or timeline (e.g., managing the presentation time of each visual element within the video). Accordingly, the video content can be edited to achieve a quality that meets the user's needs.

[0151]

[0152] FIG. 4 is a block diagram showing configurations of a video generation device according to various embodiments of the present disclosure.

[0153] The embodiment of FIG. 4 can be combined with the embodiments of FIGS. 1 to 3, or the embodiments of FIGS. 5 to 11.

[0154] Referring to FIG. 4, the content block generation module (610) can collect corporate content or corporate information from the database (300). For example, corporate content or corporate information may include corporate BI information or corporate business domain information. Corporate content or corporate information may be information with various multi-modality characteristics, such as web pages, documents, images, videos, or audio.

[0155] The content block creation module (610) can classify corporate-related content or corporate-related information into corporate BI information, corporate business domain information, or other corporate information, and extract objects through multi-modal analysis for each piece of information to create a content block.

[0156] The generated content block can be converted into a high-dimensional vector whose modality is adjusted by the embedding module and stored in the vector DB (630).

[0157] The embedding module can extract and analyze data from corporate-related content or corporate-related information, convert it into modality-adjusted high-dimensional vectors, and store it in a vector DB (630).

[0158] A user can input a user query into the video generation application (210). For example, the user may input the query, "Create a 20-second promotional video for our company's new product." The input query can be transmitted to the scenario generation module (620) of the automatic video generation device (100).

[0159] The scenario generation module (620) of the automatic video generation device (100) can convert an input user query into a vector through an embedding module, perform a similarity search with the vector for corporate data, and generate a plurality of scenarios based on a plurality of corporate data having a similarity score greater than a threshold value. Each of the plurality of scenarios includes information on the configurations of video content (e.g., the flow of the video and objects included in the video) for visually / audioally providing information included in the user query, and this can be provided to the user through the video application (210).

[0160] The video application (210) can convert multiple scenarios into videos and output them to the user, or output text information about multiple scenarios to the user.

[0161] Through the video creation application (210), the user can select a scenario that best suits the user's video production purpose or that the user prefers from among multiple scenarios, and information about this can be transmitted to the scenario creation module (620). At this time, the user can edit the video content compositions of the selected scenario, and information about the edited video content compositions can be transmitted to the scenario creation module (620).

[0162] The scenario generation module (620) can transmit information about a user query and a user-selected scenario to the LLM (640) through the vector DB (630).

[0163] LLM (640) can generate a script based on information about the input user query and the user-selected scenario. The generated script can be transmitted to the video generation application (210). The video generation application (210) can generate video content based on the script.

[0164]

[0165] FIG. 5 is a flowchart illustrating a method of generating content blocks and captions for content blocks using LLM in a service server, and extracting candidate content blocks for user queries using the captions, according to various embodiments of the present disclosure.

[0166] A content creation service server according to an embodiment of the present invention can extract main objects from corporate information including corporate content, for example, each frame of an image or video, divide them into semantic units, and create captions as metadata using a visual-language model (VLM) or an LLM that performs the function of a VLM based on information obtained from object extraction and semantic unit division.

[0167] Although not illustrated in FIG. 5, the service server can divide arbitrary content into multiple content blocks. The content block generation unit can identify the logical flow of arbitrary content, the change points in the storyline, and divide the content based on the change points to create content blocks. Alternatively, the content can be divided and created based on the rate of change in scene change or camera angle, the change points in the background or setting, the change points in the speaker or characters, or the change points in the format or style. As another example, the content block generation unit can divide the content and create content blocks based on the points where the mood or emotional line of the video significantly changes, the points where structural elements of the video such as the intro, main text, and conclusion change, the points where subtitles or overlay text change, and the points where background music or sound effects change.

[0168] The service server can then generate captions for content blocks using VLM or LLM. Content block captions can include text providing a brief description or summary of each segment or block of video.

[0169] More specifically, in step 501, the service server can extract a representative image of a content block or a representative image feature of a content block.

[0170] In step 502, the service server may generate a first prompt that says, "Generate a question about the content block and generate a response to the question in a step-by-step manner." In step 505, the service server may apply the first prompt to the LLM while transmitting the content block and a representative image of the content block to the LLM trained to understand images or videos.

[0171] Thereafter, the service server may generate a second prompt in step 507, “Generate a caption for the content block using the generated query response set,” and apply the second prompt to the LLM in step 508.

[0172] In step 510, the service server can store the caption for the content block generated by the LLM in a database by linking it with the content block as metadata for the content block.

[0173] In step 520 thereafter, when the service server receives a user query, it can extract candidate content blocks based on the similarity between the content block caption and the user query vector.

[0174] Although not shown in Figure 5, the service server can use the extracted candidate content blocks to generate other content for a user query.

[0175]

[0176] FIGS. 6 to 8 are flowcharts for explaining an operation method of an automatic video generation device according to various embodiments of the present disclosure.

[0177] The embodiments of FIGS. 6 to 8 can be combined with the embodiments of FIGS. 1 to 8.

[0178] The embodiments illustrated in FIGS. 6 to 8 may be applied to the descriptions of the configurations of embodiments in other drawings, or to the descriptions of the operations of the configurations.

[0179] Referring to FIG. 6, the operating method of the automatic video generation device may include a step (S411) of receiving corporate information (e.g., corporate content information including a web page, a document, an image, or a video); a step (S412) of extracting corporate brand identity data (e.g., a vector including corporate BI information) or corporate business domain data from the corporate information; a step (S413) of receiving a user query and extracting query data (e.g., a query vector); and a step (S414) of generating a video script based on the query data and the corporate BI data by using a learned video generation model (e.g., the video generation model (111) of FIGS. 1 and 2 or its components) that uses the corporate BI data as input.

[0180] Referring to FIG. 7, the operating method of the automatic video generation device may include a step of extracting video information from corporate information (e.g., corporate content information including a web page, a document, an image, or a video) (S421); a step of analyzing the video information to generate a content block including object information and caption information (S422); a step of analyzing the content block to generate content block data (e.g., a content block vector) (S423); a step of receiving a user query from a user device (S424); a step of analyzing the user query to generate query data (e.g., a query vector) (S425); a step of searching for a content block with the highest similarity based on the content block data and the query data using a video generation model trained with the content block data as input (e.g., the video generation model (111) of FIGS. 1 and 2 or its components) (S426); and a step of transmitting information on the searched content block to the user device (S427).

[0181] Referring to FIG. 8, the operating method of the automatic video generation device may include a step of receiving corporate information (S431); a step of extracting corporate data (e.g., corporate vector, corporate BI vector, or corporate business domain vector) from the corporate information (S432); a step of receiving a user query and extracting query data (e.g., query vector) (S433); a step of extracting user past behavior data (S434); and a step of generating reference video information based on the query data and the past behavior data by using a video generation model trained with the corporate data as input (e.g., the video generation model (111) of FIGS. 1 and 2 or its components), and transmitting the generated reference video information to a user device (S435).

[0182]

[0183] FIG. 9 is a block diagram illustrating a user device according to various embodiments of the present disclosure.

[0184] The embodiment of FIG. 9 can be combined with the embodiments of FIGS. 1 to 8.

[0185] Referring to FIG. 9, the user device (200) may include a system memory (801) including an operating system (802), a processing unit (803), a storage device (804), an input device (805), an output device (806), or a communication device (807).

[0186] The user device (200) may include a computing device or a smart device.

[0187] The system memory (801) may include random access memory (RAM) and may provide temporary storage space where the operating system and program modules are executed. The system memory (801) may temporarily store data and instructions and enable the processing unit (803) to quickly access and process data.

[0188] The operating system (802) may include software. The operating system includes memory management, file system management, processor management, and device management, and may control programs running within the device. The operating system (802) may include program modules, which are software modules for performing specific functions and may provide or be responsible for a user interface or network.

[0189] The processing unit (803) may include, but is not limited to, a CPU. The processing unit (803) may execute instructions stored in the system memory (801), process input data, and generate results or perform calculation tasks.

[0190] The storage device (804) may include a device that permanently stores data and programs.

[0191] The input device (805) is a device that receives data from a user or another system, and may include a keyboard, mouse, touch screen, or soft keys implemented on a display panel including a touch panel.

[0192] The output device (806) is a device that transmits data processed through the processing unit (803) to a user or another system, and may include a monitor, speaker, or display panel.

[0193] The communication device (807) can enable the user device (200) to be communicatively connected to an external network or another device (e.g., an automatic video generation device (100)).

[0194]

[0195] According to various embodiments of the present disclosure, a video automatic generation device may be coupled with a computer (or computing device) as hardware and may include a computer program stored in a computer-readable recording medium to perform the operations described with reference to the drawings described above as examples.

[0196] Additionally, the automatic video generation device may be implemented as a computing device including at least one processor that executes instructions of programs loaded into a memory, and a program including instructions described to execute the operations described with reference to the drawings as examples may be loaded into the memory.

[0197] The devices described above may be implemented as hardware components, software components, and / or a combination of hardware components and software components. For example, the devices and components described in the embodiments may be implemented using one or more general-purpose computers or special-purpose computers, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding to them. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. Furthermore, the processing device may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device is sometimes described as being used singly; however, those skilled in the art will appreciate that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, the processing device may include multiple processors, or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are also possible.

[0198] Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing device to perform a desired operation or may, independently or collectively, command the processing device. The software and / or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium or device, or transmitted signal wave, for interpretation by the processing device or for providing instructions or data to the processing device. The software may also be distributed over networked computer systems and stored or executed in a distributed manner. The software and data may be stored on one or more computer-readable recording media.

[0199] The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program commands, data files, data structures, etc., alone or in combination. The program commands recorded on the medium may be those specially designed and configured for the embodiment or may be those known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program commands, such as ROMs, RAMs, and flash memories. Examples of the program commands include not only machine language codes generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiment, and vice versa.

[0200] Although the embodiments described above have been described by way of limited examples and drawings, those skilled in the art will appreciate that various modifications and variations can be made based on the above teachings. For example, appropriate results can still be achieved even if the described techniques are performed in a different order than described, and / or components of the described systems, structures, devices, circuits, etc. are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents.

[0201] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims set forth below.

[0202] Although the detailed description of this document has described specific embodiments, it will be apparent to those skilled in the art that various modifications are possible without departing from the scope of this document.

Claims

1. In the operating method of the automatic video generation device, Steps to receive corporate information; A step of extracting corporate data from the above corporate information; A step of receiving a user query and extracting query data; A step of extracting the user's past behavior data; and A step of generating reference video information based on the query data and the past behavior data using a video generation model learned as input from the corporate data and transmitting the generated reference video information to a user device, How the video auto-generation device works.

2. In paragraph 1, The above reference video information is, Contains information about video templates and information about video content styles. How the video auto-generation device works.

3. In paragraph 2, The above reference video information is, Includes more information about sample video content, How the video auto-generation device works.

4. In paragraph 1, If the similarity between the query data and the past behavior data is greater than a specified threshold value, the reference video information includes reference video information included in the past behavior data. How the video auto-generation device works.

5. In paragraph 1, A step of generating a prompt based on the above query data; A step of generating a context based on reference video information selected by the user; and A step of generating a video script based on the above prompt, the above query data, and the above context, How the video auto-generation device works.

6. In paragraph 5, A step of generating video content based on the above video script; A step of transmitting the above video content to a user device; and Further comprising a step of receiving completion information or modification information entered by the user, How the video auto-generation device works.

7. In paragraph 6, In case of receiving modification information by the user, the method further includes a step of reproducing the video content by modifying at least one of object information and background information of the video content based on the modification information and transmitting the reproducing video content to the user device. How the video auto-generation device works.

8. In the automatic video generation device, at least one processor; and Contains memory that stores instructions, The above instructions, when individually or collectively executed by the at least one processor, cause the processor to: Receive corporate information, Extract corporate data from the above corporate information, Receive user queries and extract query data, Extract the user's past behavior data, Using a video generation model learned with the above corporate data as input, reference video information is generated based on the query data and the past behavior data and transmitted to the user device. Automatic video generation device.

9. Combined with hardware, stored in a computer-readable recording medium to perform the operation method of the automatic video generation device of any one of claims 1 to 7. Computer program.

10. In the automatic video generation system, A user device that receives user queries; and Includes a video auto-generation device, The above video automatic generation device extracts corporate data from the corporate information, receives a user query and extracts query data, extracts the user's past behavior data, and uses a video generation model learned with the corporate data as input to generate reference video information based on the query data and the past behavior data and transmits the generated reference video information to a user device. Automatic video generation system.

Images