VFX generation intention analysis system using LLM, and method thereof

Abstract

A VFX generation intention analysis system using an LLM, according to the present invention, comprises: a data collection module, which collects source data for generating training data; a training data generation module, which generates training data for generating a user-customized LLM by fine-tuning a general-purpose LLM by using data from the data collection module; an LLM training module for fine-tuning a user-customized LLM model by using the training data from the training data generation module; and a system prompt setting module capable of generating, storing, and applying a system prompt for following a predetermined output structure of the LLM training module and generating results of a predetermined quality.

Description

VFX Creation Intent Analysis System and Method Using LLM

[0001] The present invention relates to a system and method for analyzing the intent to create Visual Effects (VFX) using a Large Language Model (LM), and more specifically, to a system and method for analyzing what intent a user has and what VFX effect they intend to create, and transmitting this information to an AI model.

[0002] Visual Effects (VFX) is a technology used in various media such as movies, television, commercials, and video games to create scenes that are difficult to express realistically. It is generally used to realize imaginary scenes, characters, and environments on screen by combining live-action filming with computer graphics.

[0003] Currently, VFX production is primarily carried out manually by experts, requiring high technical skills, significant time, and considerable effort to realize creative ideas. This acts as a factor that significantly increases the production costs and time of video content, and there are limitations in rapidly and efficiently implementing various visual effects.

[0004] Generative AI is a field of artificial intelligence technology consisting of algorithms capable of generating new content by learning from given data; by utilizing this, various types of data such as text, images, and videos can be produced. Recently, research on video generation using AI has become significantly more active following OpenAI's release of Sora, a Text-to-Video model.

[0005] Prompt engineering is essential for producing high-quality VFX effects using generative AI, and this work is based on the trial and error and know-how of prompt engineers. This requires various abilities, such as an understanding of the AI ​​model to be used, vocabulary, and creativity, which acts as a kind of barrier to entry for users.

[0006] While efforts are ongoing to effectively utilize video generation AI models, similar to generative AI models, no technology has yet been disclosed in the VFX field that analyzes user input text to determine the user's creation intent and generates prompts to be passed to an AI model that creates VFX effects based on this analysis.

[0007] Large Language Models (LLMs) are artificial intelligence models trained on vast amounts of text data that demonstrate excellent performance in understanding and generating natural language. Representative commercial services utilizing LLMs include OpenAI's ChatGPT and Google's Gemini.

[0008] The present invention has been devised in response to the aforementioned necessity and aims to provide an analysis system and method that utilizes LLM to analyze user input data to identify user intent, automatically generates input prompts for use by generative AI, and provides a service for delivering these prompts in the format required by the generative AI.

[0009] A VFX generation intent analysis system utilizing LLM according to the present invention for the above technical problem comprises: a data collection module for collecting source data for generating training data; a training data generation module for generating training data for generating a user-customized LLM by fine-tuning a general-purpose LLM (Large Language Model) using the data from the data collection module; an LLM learning module for fine-tuning a user-customized LLM model using the training data from the training data generation module; a system prompt setting module capable of generating, saving, and applying a system prompt to produce results of a certain quality while following a predetermined output structure of the LLM learning module; a user intent specification module for obtaining necessary information from a user by applying the system prompt setting module; and a user intent analysis module for classifying the category of VFX (Visual Effects) that the user intends to generate from the data of the user intent specification module and generating a prompt for generating VFX effects to be delivered to a generative AI.

[0010] Fine-tuning of an LLM learning model is characterized by adjusting parameters including batch size, epoch, and learning rate.

[0011] The above user intent specification module is characterized by utilizing interactive LLM to engage in conversation with the user until all pre-entered essential information is gathered, asking the user what thoughts they have regarding each item, obtaining the answers, and collecting them all.

[0012] The above user intent analysis module is characterized by generating and outputting prompts in sentence and keyword formats and transmitting them to a generative AI model.

[0013] It is characterized by including a data conversion module that converts the generated prompt into a form required by the generative AI.

[0014] The above data conversion module is characterized by converting into a CLIP vector.

[0015] It is characterized by including a data repository that stores data necessary for generating training data and fine-tuned user-customized LLMs.

[0016] The above-described training data generation module is characterized by including specialized terminology in the VFX field and manual information on major tools, and generating training data by utilizing reverse prompt engineering techniques.

[0017] The VFX generation intent analysis system utilizing LLM according to the present invention can reduce video production costs and time by automatically identifying what kind of VFX effect a creator intends to make simply by inputting text, and automatically generating prompts for producing high-quality results, so that VFX effects currently produced manually by experts can be automatically generated using video generation AI.

[0018] Furthermore, if the general public can easily produce VFX effects by utilizing VFX creation intent analysis technology and video generation AI, they can contribute to the creation of a healthy prosuming ecosystem by sharing or selling numerous VFX effects that highlight diversity and creativity.

[0019] In addition, the present invention can be generalized and abstracted to enable universal utilization, allowing it to be applied to fields other than VFX generation.

[0020] Figure 1 is an overall configuration diagram of a VFX generation intent analysis system utilizing LLM according to the present invention.

[0021] FIG. 2 is a block diagram of a service server configuration according to the present invention.

[0022] FIG. 3 is an example flowchart of a user intent realization module according to the present invention.

[0023] FIG. 4 is an example flowchart of a user intent analysis module according to the present invention.

[0024] Figure 5 is an overall flowchart of a method for analyzing VFX generation intent using LLM according to the present invention.

[0025] FIG. 6 is a flowchart of the reverse prompt engineering-based learning data generation flowchart according to the present invention.

[0026] FIG. 7 is a flowchart of a method for realizing a user's intention according to the present invention.

[0027] FIG. 8 is a flowchart of a user intent analysis method according to the present invention.

[0028] FIG. 1 is an overall configuration diagram of a VFX generation intent analysis system utilizing LLM according to the present invention, comprising a service server and a user terminal. The service server includes a plugin module that provides VFX production tools, an LLM module which is a generation AI model, and an API (application programming interface) module that interacts with applications. When used independently, the user terminal downloads and uses the plugin module and the LLM module from the service server, and when API communication is possible, communicates with the API module through the plugin module.

[0029] FIG. 2 is a block diagram of a service server configuration, comprising: a data collection module for collecting source data for generating training data; a training data generation module for generating training data to create a custom LLM by fine-tuning a general-purpose LLM; an LLM learning module for fine-tuning a custom LLM model to be used in the VFX effect classification module and input prompt generation module, which are user intent analysis steps; a system prompt setting module for generating / saving / applying system prompts for the LLM to follow a defined output structure and generate results of a certain quality; a user intent specification module for obtaining necessary information from the user by applying the system prompt setting module; and a user intent analysis module for classifying the category of VFX effects that the user intends to generate based on user intent and generating prompts for generating VFX effects to be delivered to the generative AI.

[0030] It further includes a data conversion module that converts the generated prompt into the form required by the generative AI.

[0031] It further includes a data store that stores the data required to generate training data and fine-tuned user-customized LLMs.

[0032] The above data collection module collects specialized terminology in the VFX field and manual information for major VFX tools from websites, etc., and can obtain predefined essential information items from users through a chat interface via expert interviews, etc.

[0033] The aforementioned training data generation module generates training data by utilizing specialized terminology in the VFX field, manual information from major tools, and the Reverse Prompt Engineering technique. Reverse Prompt Engineering is a process of analyzing generated output to infer or reconstruct the prompt that produced that output in reverse.

[0034] The above LLM training module is based on the pretrain module of a general-purpose LLM, performs fine-tuning using training data generated through the training data generation module, and creates and saves a Custom LLM. During fine-tuning, various values ​​can be entered for parameters such as batch size, epoch, and learning rate.

[0035] The aforementioned system prompt configuration module performs prompt engineering to enable the LLM operating in each module to produce results suitable for the purpose, stores system prompts designed to generate high-quality results, and allows each module to specify which system prompt to use. Through this module, VFX effect classifications are predefined in the LLM, and the LLM is configured to export high-quality results in the desired format.

[0036] A prompt is an input given to an AI model that specifies the direction or content of the content the model will generate, and prompt engineering refers to the process of designing and optimizing prompts to obtain desired results in generative AI models.

[0037] The above user intent concretization module utilizes an interactive LLM to converse with the user until all pre-entered essential information is gathered, and obtains answers by asking the user what they think about each item. Once all information is gathered, it aggregates all the information obtained through the conversation and transmits it to the user intent analysis module. Figure 3 is an example of a user intent concretization module.

[0038] The user intent analysis module described above utilizes an LLM to analyze user input data and classifies and outputs which VFX effect the user intends to create from among predefined VFX effect categories. Furthermore, based on the results of analyzing the user input data using the LLM, it generates and outputs input prompts in sentence and keyword formats necessary to create the desired VFX effect through a generative AI. While the present invention primarily uses a Custom LLM, a general-purpose LLM may also be used.

[0039] Figure 4 illustrates an example of a user intent analysis module that performs analysis and generates results in two stages. First, to ensure that a generative AI model optimized for each category generates VFX effects, the text entered by the user is classified to determine which category of VFX effect is intended to be generated. Second, a prompt to be sent to the VFX generative model is generated based on the text entered by the user. For example, if the user inputs, "Create an effect where a passenger car explodes due to a terrorist attack in the middle of the city," this is first analyzed to derive the VFX effect category "Explosion." Then, a prompt consisting of "City, Terror, Passenger Car, Explosion" is generated, and a request to generate the VFX effect is sent to a generative model specialized in creating explosion effects. To implement this, various templates and system prompts are tested, and those with good quality results are selected to perform additional prompt engineering, thereby improving the quality of the results. A configuration template is provided to allow the user to set several items, such as creativity for prompt generation.

[0040] The above data conversion module converts the data obtained by the user intent concretization module into the format required by the user intent analysis module. In addition, it aggregates the data output by the VFX effect classification module and the input prompt generation module of the user intent analysis module and converts the data into the format required by the generative AI that will receive the result generated by the present invention. Basically, it converts into a CLIP (Contrastive Language-Image Pre-training) vector. CLIP is an embedding model developed by OpenAI, which is a model trained to vectorize text data and image data so that text and images with the same meaning have similar vector values ​​in the same vector space, and a CLIP vector is data vectorized through the CLIP embedding model.

[0041] The above data repository stores various source data used for generating training data and the generated training data, and provides data to the training data generation module. It also stores the fine-tuning Custom LLM generated by the LLM training module and provides LLM data to the VFX effect classification module and the input prompt generation module.

[0042] The present invention is distinguished by a module capable of automatically generating training data for LLM model training, a module capable of fine-tuning basic LLM, specialization in the VFX field, support for acquiring necessary information and concretizing ideas through conversation with the user, a module for setting and classifying categories of VFX effects that the user can generate, and a module for converting data into a format required by generative AI.

[0043] When a user who wishes to create VFX effects inputs the desired effect as text, the interactive LLM derives the necessary information through dialogue with the user. Based on this, it utilizes the LLM to analyze user intent, classify VFX effects, and automatically generate prompts for VFX creation. Finally, it converts the data into the format required by the generative AI to be linked later and delivers it.

[0044] This invention enables a conversational LLM to obtain necessary information through dialogue between a user intending to generate VFX effects, thereby allowing the user to concretize their intentions and ideas. Based on this acquired information, a Custom LLM classifies the specific VFX effects the user wishes to create and generates prompts appropriate to the intent. The generated data is aggregated and converted into a format according to the specifications of a generative AI model to be linked later, after which the data is transmitted. While primarily specialized for the VFX field, it can be expanded and applied to other fields by creating additional Custom LLMs and changing the linked generative AI model in the future.

[0045] Figure 5 is an overall flowchart of a method for analyzing VFX generation intent using LLM according to the present invention.

[0046] FIG. 6 is a flowchart of the Reverse Prompt Engineering-based learning data generation flowchart utilizing specialized terminology and related data in the VFX field according to the present invention.

[0047] FIG. 7 is a flowchart of a method for specifying user intent using LLM according to the present invention.

[0048] FIG. 8 is a flowchart of a Custom LLM-based user intent analysis method according to the present invention.

[0049] The VFX generation intent analysis system utilizing LLM according to the present invention can be applied to a service that analyzes user input data using LLM to identify user intent, automatically generates input prompts for use by a generative AI, and delivers them in the format required by the generative AI.

[0050] In addition, VFX effects can be shared or sold, and the invention can be generalized and abstracted to enable universal utilization, allowing it to be applied to fields other than VFX creation.

Claims

1. A data collection module that collects source data for generating training data; A training data generation module that generates training data for creating a user-customized LLM by fine-tuning a general-purpose LLM (Large Language Model) using data from the above data collection module; An LLM learning module that fine-tunes a user-customized LLM model using the training data of the above-mentioned training data generation module; A system prompt setting module capable of creating, saving, and applying system prompts to follow the defined output structure of the above LLM learning module and generate results of a certain quality; A user intent specification module that obtains necessary information from the user by applying the above-mentioned system prompt setting module, and A VFX generation intent analysis system utilizing LLM, characterized by including a user intent analysis module that classifies the category of VFX (Visual Effects) to be generated by the user from the data of the user intent specification module and generates a prompt for generating VFX effects to be delivered to a generative AI.

2. In Paragraph 1, A VFX generation intent analysis system utilizing LLM, characterized by fine-tuning the LLM learning model by inputting parameters including batch size, epoch, and learning rate.

3. In Paragraph 1, The above user intent specification module is a VFX generation intent analysis system utilizing LLM, characterized by using interactive LLM to converse with the user until all pre-entered essential information is gathered, asking the user what thoughts they have regarding each item, obtaining the answers, and collecting them all.

4. In Paragraph 1, A VFX generation intent analysis system utilizing LLM, characterized by the above-mentioned user intent analysis module creating and outputting prompts in sentence format and keyword format and transmitting them to a generative AI model.

5. In Paragraph 1, A VFX generation intent analysis system utilizing LLM, characterized by including a data conversion module that converts a generated prompt into a form required by a generative AI.

6. In Paragraph 5, A VFX generation intent analysis system utilizing LLM characterized by the above data conversion module converting into CLIP vectors.

7. In Paragraph 1, A VFX generation intent analysis system utilizing LLM, characterized by including a data repository that stores data necessary for generating training data and fine-tuned user-customized LLM.

8. In Paragraph 1, A VFX generation intent analysis system utilizing LLM, characterized by the above-mentioned learning data generation module including specialized terminology in the VFX field and manual information of major tools, and generating learning data using reverse prompt engineering techniques.

Images