Video generation system and method based on intelligent agent

Abstract

The invention discloses a video generation system and method based on an intelligent agent, and relates to the technical field of video generation, and the system comprises a user agent agent, an assistant agent and an external function library. The user agent intelligent agent and the assistant intelligent agent are intelligent agents based on a large language model; the user agent intelligent agent and the assistant intelligent agent import functions in the external function library into the function library of the user agent intelligent agent and the assistant intelligent agent; the user agent intelligent body is used for sending a character video requirement of a user to the assistant intelligent body; the assistant agent is used for selecting a function from the function library according to a character video requirement, or an execution result returned by the user agent agent or a next working command, and sending the selected function and corresponding parameters to the user agent agent until a final video is generated; and the user agent agent is used for executing the function selected by the assistant agent and returning an execution result of the function to the assistant agent. The video generation quality can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of video generation, and in particular to a video generation system and method based on intelligent agent. Background Art

[0002] With the rise of Large Language Models (LLM) such as ChatGenerative Pre-trained Transformer (ChatGPT), users have a growing demand for fast generation of personalized videos. However, LLM has limited built-in functions and cannot independently complete highly complex and customized tasks, making it difficult to meet the needs of professional-level video generation, resulting in low efficiency in handling complex tasks and inaccurate video generation results. Summary of the invention

[0003] The purpose of this application is to provide a video generation system and method based on intelligent agent, which can improve the quality of video generation.

[0004] To achieve the above objectives, this application provides the following solutions:

[0005] In a first aspect, the present application provides a video generation system based on an intelligent agent, the video generation system based on an intelligent agent comprising: a user agent agent, an assistant agent, and an external function library; the user agent agent and the assistant agent are both agents based on a large language model; the user agent agent and the assistant agent both import functions in the external function library into their own function library;

[0006] The user agent agent is used to send the user's text video request to the assistant agent;

[0007] The assistant agent is used to select a function from a function library according to the text and video requirements, the execution result returned by the user agent agent, or the next work command, and send the selected function and corresponding parameters to the user agent agent;

[0008] The user agent agent is used to execute the function selected by the assistant agent and return the execution result of the function to the assistant agent;

[0009] The assistant agent is also used to determine whether to continue executing the function based on the execution result returned by the user agent agent. If so, continue to select the function from the function library based on the execution result currently returned by the user agent agent. Otherwise, the user agent agent determines whether to execute the next step until the final video is generated. If the user agent agent determines to execute the next step, the next step command is sent to the assistant agent.

[0010] Optionally, the intelligent agent-based video generation system also includes an intelligent agent framework, which is used to record virtual dialogue content between the user agent agent and the assistant agent, and the virtual dialogue content includes function call links.

[0011] Optionally, the intelligent agent-based video generation system also includes a result output module; the result output module is used to output the final answer, function directory and function call link of the assistant intelligent agent, the final answer is the final video generated, and the function directory is a function directory composed of functions executed by the user agent intelligent agent in the process of generating the final video.

[0012] Optionally, the intelligent agent-based video generation system further includes a title acquisition module, and the title acquisition module is used to acquire text video requirements input by the user and a selected large language model.

[0013] Optionally, the large language model is ChatGPT, LLaMA, Doubao Large Model or Tongyi Qianwen.

[0014] Optionally, the external function library includes an information crawling module, a text-to-speech module, a template editing module, a template filling module, a mouth shape alignment module and a video enhancement module.

[0015] In a second aspect, the present application provides a video generation method based on an intelligent agent, the video generation method based on an intelligent agent comprising:

[0016] The user agent sends the user's text video request to the assistant agent; the user agent and the assistant agent are both agents based on a large language model; the user agent and the assistant agent both import the functions in the external function library into their own function libraries;

[0017] The assistant agent selects a function from a function library according to the text and video requirements, or the execution result or next work command returned by the user agent agent, and sends the selected function and corresponding parameters to the user agent agent;

[0018] The user agent agent executes the function selected by the assistant agent and returns the execution result of the function to the assistant agent;

[0019] The assistant agent determines whether to continue to execute the function according to the execution result returned by the user agent agent. If yes, the assistant agent continues to select a function from the function library according to the execution result returned by the current user agent agent, and sends the selected function and corresponding parameters to the user agent agent, returns the user agent agent to execute the function selected by the assistant agent, and returns the execution result of the function to the assistant agent. Otherwise, the user agent agent determines whether to execute the next step. If the user agent agent determines to execute the next step, the next step command is sent to the assistant agent.

[0020] The assistant agent selects a function from the function library according to the next work command, and sends the selected function and corresponding parameters to the user agent agent, and returns the user agent agent to execute the function selected by the assistant agent, and returns the execution result of the function to the assistant agent until the final video is generated.

[0021] Optionally, the intelligent agent-based video generation method further includes: recording virtual conversation content between the user agent agent and the assistant agent through an intelligent agent framework, wherein the virtual conversation content includes a function call link.

[0022] Optionally, after generating the final video, the intelligent agent-based video generation method further includes:

[0023] Output the assistant agent's final answer, function directory and function call link, wherein the final answer is the generated final video, and the function directory is a function directory consisting of functions executed by the user agent in the process of generating the final video.

[0024] Optionally, before the user agent sends the user's text video requirements to the assistant agent, the intelligent agent-based video generation method further includes: obtaining the text video requirements input by the user and the selected large language model.

[0025] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0026] The present application provides a video generation system and method based on intelligent agents. The user agent agent and the assistant agent are both agents based on a large language model. The user's text video requirements are converted into structured instructions by means of a virtual dialogue between the user agent agent and the assistant agent, thereby achieving the decomposition and refinement of the text video requirements, making full use of external function libraries, and improving the quality of video generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0028] Figure 1 A schematic diagram of a workflow of a video generation system based on an intelligent agent provided in one embodiment of the present application;

[0029] Figure 2 A schematic diagram of the collaboration process of each module in an external function library provided in one embodiment of the present application. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0031] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0032] The present application provides a video generation system based on intelligent agents, which includes: a user agent agent, an assistant agent, and an external function library; the user agent agent and the assistant agent are both agents based on a large language model. The computing power of the user agent agent and the assistant agent comes from the large language model; the user agent agent and the assistant agent both import functions in the external function library into their own function library.

[0033] The user agent agent is used to send the user's text and video requirements to the assistant agent.

[0034] The assistant agent is used to select a function from a function library according to the text and video requirements, the execution result returned by the user agent agent or the next work command, and send the selected function and corresponding parameters to the user agent agent.

[0035] The user agent agent is used to execute the function selected by the assistant agent and return the execution result of the function to the assistant agent.

[0036] The assistant agent is also used to determine whether to continue executing the function based on the understanding ability of the large language model according to the execution result returned by the user agent agent. If so, continue to select functions from the function library according to the execution result currently returned by the user agent agent. Otherwise, the user agent agent determines whether to execute the next step until the final video is generated. If the user agent agent determines to execute the next step, the next step command is sent to the assistant agent.

[0037] The user agent agent is used to control the start and end of the virtual dialogue interaction between the assistant agent and the user agent agent.

[0038] In this application, the user agent agent and assistant agent generate the final video based on the user's text video requirements through virtual dialogue interaction.

[0039] The user agent and assistant agent of the present application are both agents based on a large language model. The user's text and video requirements are converted into structured instructions by means of a virtual dialogue between the user agent and the assistant, thereby achieving the decomposition and refinement of the text and video requirements, making full use of the external function library, and improving the quality of video generation.

[0040] like Figure 1 As shown, the workflow of a video generation system based on intelligent agent in this application is as follows:

[0041] Step 101: Obtain the video requirements and selected LLM input by the user, initialize the system and create two agents: a user agent agent and an assistant agent, and provide the external function library for the two agents to access.

[0042] The code for the intelligent agent to create an assistant is shown in Table 1.

[0043] Table 1 Creating an assistant agent

[0044]

[0045] The code for the intelligent agent to create a user agent agent (user_proxy) is shown in Table 2.

[0046] Table 2 Creating an Assistant Agent

[0047]

[0048]

[0049] After creating the user agent and assistant agent, we start registering functions. First, we import the external function library and register the functions in the external function library to the user agent and assistant agent. In this way, the assistant agent can call these functions when needed, and the user agent is responsible for executing and returning the results.

[0050] Step 102: The user agent initiates a conversation and sends the user's text video request to the assistant agent for analysis.

[0051] Step 103: The assistant agent analyzes and understands the information provided by the user agent agent, and automatically selects appropriate functions and parameters from the external function library as required, and sends them to the user agent agent for execution.

[0052] Step 104: The user agent executes the function and parameters provided by the assistant agent, and provides the function return value to the assistant agent. The assistant agent determines whether it needs to continue executing the function. If so, the function is executed according to the method of steps 103 and 104; otherwise, the user agent determines the next step.

[0053] The user agent of this application initiates a conversation based on the video requirements input by the user. The assistant agent selects appropriate functions and parameters based on the content of the conversation, and the user agent executes these functions and returns the results. The entire conversation process and the function links called will be recorded to facilitate technical personnel to debug and optimize. The assistant agent automatically selects functions from the defined external function library to execute and process the content of the question based on the text video requirements.

[0054] Step 105: Use the intelligent agent framework to record the virtual conversation content between the assistant agent and the user agent agent, where the virtual conversation content includes available functions and function call links.

[0055] Step 106: The user agent detects that the assistant agent has completed video generation and ends the virtual conversation; the intelligent agent framework outputs the generated final video, function directory and function call link.

[0056] In an exemplary embodiment, the intelligent agent-based video generation system further includes an intelligent agent framework, which is used to record the virtual dialogue content between the user agent agent and the assistant agent, wherein the virtual dialogue content includes a function call link. When the system is initialized, the intelligent agent framework converts the external function library into a function library that can be executed by the agent, and provides it to the agent for access while initializing the agent.

[0057] In an exemplary embodiment, the intelligent agent-based video generation system also includes a result output module; the result output module is used to output the final answer, function directory and function call link of the assistant intelligent agent, the final answer is the final video generated, and the function directory is a function directory composed of functions executed by the user agent intelligent agent in the process of generating the final video.

[0058] In an exemplary embodiment, the intelligent agent-based video generation system also includes a title acquisition module, which is used to obtain the text video requirements input by the user and the selected large language model after the system is initialized, and provide the text video requirements and the large language model to the user agent intelligent body to start analysis.

[0059] In an exemplary embodiment, the large language model is a large language model such as ChatGPT, LLaMA, Doubao Large Model or Tongyi Qianwen.

[0060] In an exemplary embodiment, the external function library includes an information crawling module, a text-to-speech (TTS) module, a template editing module, a template filling module, a mouth alignment module and a video enhancement module.

[0061] In an exemplary embodiment, the video generation process includes the following steps: generating director's words according to the user's text video requirements; generating audio according to the director's words; generating a video template according to the audio; generating a video draft according to the video template; lip aligning the video draft; enhancing the video after lip alignment to generate a final video draft, i.e., the final video.

[0062] The following is a specific example to describe the workflow of the video generation system based on intelligent agents in the present application. The video generation system based on intelligent agents in the present application is used to complete the following tasks: the user needs to generate a customized digital human advertising video for a product. The title sent, that is, the text video requirement is: I want to generate a female digital human broadcast video. The overall product content is based on this link: PRETTYGARDEN Women's Summer Tie One Shoulder BohoFloral Dress Elastic Waist Tiered Ruffle ALine Flowy Mini DressesatAmazonWomen's Clothing store. The template and background are required to be light and active, the language is English, and the target audience is young women in California, USA.

[0063] To this end, it is necessary to call the URL TO VIDEO function library, which is an external function library. The functions included in this function library can help users generate digital human advertising videos based on the shopping website's Uniform Resource Locator (URL). The process of each module in this function library working together to complete this work is as follows:

[0064] Step 201: The user inputs video requirements (text video requirements), selects LLM, the intelligent agent starts, and creates two agents: the user agent agent and the assistant agent; during this process, the intelligent agent automatically reads the introduced function library and converts it into a function library that can be executed by the agent, and provides it to the agent for access while initializing the agent.

[0065] Step 202: The user agent agent receives the user's video request and sends it to the assistant agent for analysis.

[0066] Step 203: The assistant agent automatically determines that an information crawling module needs to be called, the parameter is the URL link in the video request, and submits the information crawling module and the parameter to the user agent agent.

[0067] Step 204: The user agent agent executes the information crawling module according to the function (information crawling module) and parameters provided by the assistant agent and obtains the return value (the crawled product description and picture), and submits it to the assistant agent.

[0068] Step 205: The assistant agent analyzes the product description, and generates the director's words based on the language and audience requirements (English, young women in California, USA) in the user's video requirements, and submits it to the user agent agent (this process is completed by the large language model in the assistant agent. If there is a separate director's words generation module in the function library, the assistant agent will submit this module and corresponding parameters to the user agent agent).

[0069] Step 206: The user agent agent determines that audio needs to be generated based on the director's words, and issues this command to the assistant agent.

[0070] Step 207: The assistant agent determines that the TTS module is needed to generate audio, and the parameters are the director's words, a female digital human, and a brisk and active tone, and submits the above module and parameters to the user agent agent.

[0071] Step 208: The user agent agent executes the TTS module, generates the final audio draft, and submits it to the assistant agent.

[0072] Step 209: The assistant agent recognizes the above return value as the final audio draft, reports to the user agent agent that the final audio draft has been generated, and requests the next step of work.

[0073] Step 210: The user agent agent does not detect that the video is complete, determines that the video needs to be generated based on the audio, and issues this command to the assistant agent.

[0074] Step 211: The assistant agent determines that the template editing module needs to be called, and the parameters are product pictures, director's words (subtitles), audio, and selected digital human (if the user has not selected, the default digital human of the corresponding gender is used), and submits it to the user agent agent.

[0075] Step 212: The user agent agent executes the template editing module, generates a video template, and submits it to the assistant agent.

[0076] Step 213: The assistant agent determines that the template filling module needs to be called, and the parameters are the edited template, product image, director's words (subtitles), audio, and selected digital human, and submits it to the user agent agent.

[0077] Step 214: The user agent agent executes the template filling module to generate a draft video (without lip alignment) and submits it to the assistant agent.

[0078] Step 215: The assistant agent determines that the mouth alignment module needs to be called, with the parameters of the video draft and audio, and submits it to the user agent agent.

[0079] Step 216: The user agent agent executes the mouth alignment module, generates a mouth-aligned video, and submits it to the assistant agent.

[0080] Step 217: The assistant agent determines that the video enhancement module needs to be called, the parameter is the video after lip alignment, and submits it to the user agent agent.

[0081] Step 218: The user agent agent executes the video enhancement module, generates a final video draft, and submits it to the assistant agent.

[0082] Step 219: The assistant agent recognizes the above return value as the final video draft, reports to the user agent agent that the final video draft has been generated, and requests the next step of work.

[0083] Step 220: The user agent detects that the video has been completed and sends a TERMINATE command to end the session.

[0084] Step 221: The intelligent agent framework service now includes all session information, from which the last session of the assistant agent (final video draft), the first session of the user agent (user video request), all sessions containing function calls and function return values ​​(such sessions do not include other content except function name, parameters, and return value), and available function libraries are extracted. The above information is summarized and output to end.

[0085] In the process of completing this task, the intelligent agent automatically determines the workflow and determines whether to call a function, so the intelligent agent can automatically identify other modules added or switch to other function libraries, ensuring continuity and versatility. For example, in order to further improve the video quality, the video generation system based on the intelligent agent can automatically identify and use other modules or function libraries added externally, such as an external function library that also includes a director's speech quality evaluation module, an audio quality evaluation module, and a video quality evaluation module.

[0086] Repeating the above process, due to the existence of the above evaluation module, the user agent intelligent body will determine that the generated director's speech needs to be evaluated after step 206, and the assistant intelligent body will determine that the audio evaluation module needs to be continued after step 209, and the video evaluation module needs to be continued after step 215. It can be seen that the rich external function library can be automatically recognized and utilized by this application, which greatly improves the working performance of this application.

[0087] This application aims to improve the work efficiency, result accuracy and human-computer interaction convenience of LLM in processing text video tasks. Through the intelligent agent framework, LLM can automatically call external functions to improve its functionality and effectiveness in extracting text requirements and generating high-quality videos. The interaction between the two intelligent agents reduces the number of human-computer interactions. Users can get the final video draft by inputting video requirements, which improves the human-computer interaction experience and is suitable for various scenarios that require efficient content generation. This system can also synchronously report the identified function libraries and function call links to technicians, help technicians discover and solve problems, enable technicians to monitor the operation process, improve debugging efficiency, further optimize problems, and improve the quality of LLM answers.

[0088] This application can be widely used in video editing, advertisement generation, content creation and other fields to provide users with efficient and accurate content generation services.

[0089] Based on the same inventive concept, the embodiment of the present application also provides a method for generating video based on an intelligent agent for realizing the video generating system based on an intelligent agent involved above. The implementation scheme for solving the problem provided by the method is similar to the implementation scheme recorded in the above system, so the specific limitations in one or more embodiments of the video generating method based on an intelligent agent provided below can refer to the limitations of the video generating system based on an intelligent agent above, and will not be repeated here.

[0090] In an exemplary embodiment, a video generation method based on an intelligent agent is provided, and the video generation method based on an intelligent agent includes:

[0091] The user agent sends the user's text and video request to the assistant agent; both the user agent and the assistant agent are agents based on a large language model; both the user agent and the assistant agent import the functions in the external function library into their own function libraries.

[0092] The assistant agent selects a function from a function library according to the text or video requirements, or the execution result or next work command returned by the user agent agent, and sends the selected function and corresponding parameters to the user agent agent.

[0093] The user agent agent executes the function selected by the assistant agent and returns the execution result of the function to the assistant agent.

[0094] The assistant agent determines whether to continue executing the function based on the execution result returned by the user agent agent. If so, it continues to select a function from the function library based on the execution result returned by the current user agent agent, and sends the selected function and corresponding parameters to the user agent agent, returns the step of the user agent agent executing the function selected by the assistant agent, and returns the execution result of the function to the assistant agent. Otherwise, the user agent agent determines whether to execute the next step. If the user agent agent determines to execute the next step, the next step command is sent to the assistant agent.

[0095] The assistant agent selects a function from the function library according to the next work command, and sends the selected function and corresponding parameters to the user agent agent, and returns the user agent agent to execute the function selected by the assistant agent, and returns the execution result of the function to the assistant agent until the final video is generated.

[0096] In an exemplary embodiment, the intelligent agent-based video generation method further includes: recording virtual conversation content between the user agent agent and the assistant agent through an intelligent agent framework, wherein the virtual conversation content includes a function call link.

[0097] In an exemplary embodiment, after generating the final video, the intelligent agent-based video generation method further includes: outputting the final answer, function directory and function call link of the assistant intelligent agent, wherein the final answer is the generated final video, and the function directory is a function directory consisting of functions executed by the user agent intelligent agent in the process of generating the final video.

[0098] In an exemplary embodiment, before the user agent sends the user's text video requirements to the assistant agent, the intelligent agent-based video generation method also includes: obtaining the text video requirements input by the user and the selected large language model.

[0099] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0100] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A video generation system based on intelligent agent, characterized in that: The video generation system based on intelligent agent includes: a user agent intelligent agent, an assistant intelligent agent and an external function library; the user agent intelligent agent and the assistant intelligent agent are both intelligent agents based on a large language model; the user agent intelligent agent and the assistant intelligent agent both import functions in the external function library into their own function library; The user agent agent is used to send the user's text video request to the assistant agent; The assistant agent is used to select a function from a function library according to the text and video requirements, the execution result returned by the user agent agent, or the next work command, and send the selected function and corresponding parameters to the user agent agent; The user agent agent is used to execute the function selected by the assistant agent and return the execution result of the function to the assistant agent; The assistant agent is also used to determine whether to continue executing the function based on the execution result returned by the user agent agent. If so, continue to select the function from the function library based on the execution result currently returned by the user agent agent. Otherwise, the user agent agent determines whether to execute the next step until the final video is generated. If the user agent agent determines to execute the next step, the next step command is sent to the assistant agent.

2. The video generation system based on intelligent agent according to claim 1, characterized in that: The intelligent agent-based video generation system also includes an intelligent agent framework, which is used to record the virtual dialogue content between the user agent agent and the assistant agent, and the virtual dialogue content includes function call links.

3. The video generation system based on intelligent agent according to claim 2, characterized in that: The intelligent agent-based video generation system also includes a result output module; the result output module is used to output the final answer, function directory and function call link of the assistant intelligent agent, the final answer is the final video generated, and the function directory is a function directory composed of functions executed by the user agent intelligent agent in the process of generating the final video.

4. The video generation system based on intelligent agent according to claim 1, characterized in that: The intelligent agent-based video generation system also includes a topic acquisition module, which is used to obtain text video requirements input by the user and the selected large language model.

5. The intelligent agent-based video generation system according to claim 1, characterized in that: The large language model is ChatGPT, LLaMA, Doubao large model or Tongyi Qianwen.

6. The intelligent agent-based video generation system according to claim 1, characterized in that: The external function library includes an information crawling module, a text-to-speech module, a template editing module, a template filling module, a mouth shape alignment module and a video enhancement module.

7. A video generation method based on intelligent agent, characterized in that: The video generation method based on intelligent agent includes: The user agent sends the user's text video request to the assistant agent; the user agent and the assistant agent are both agents based on a large language model; the user agent and the assistant agent both import the functions in the external function library into their own function libraries; The assistant agent selects a function from a function library according to the text and video requirements, the execution result returned by the user agent agent, or the next work command, and sends the selected function and corresponding parameters to the user agent agent; The user agent agent executes the function selected by the assistant agent and returns the execution result of the function to the assistant agent; The assistant agent determines whether to continue to execute the function according to the execution result returned by the user agent agent. If yes, the assistant agent continues to select a function from the function library according to the execution result returned by the current user agent agent, and sends the selected function and corresponding parameters to the user agent agent, returns the user agent agent to execute the function selected by the assistant agent, and returns the execution result of the function to the assistant agent. Otherwise, the user agent agent determines whether to execute the next step. If the user agent agent determines to execute the next step, the next step command is sent to the assistant agent. The assistant agent selects a function from the function library according to the next work command, and sends the selected function and corresponding parameters to the user agent agent, and returns the user agent agent to execute the function selected by the assistant agent, and returns the execution result of the function to the assistant agent until the final video is generated.

8. The video generation method based on intelligent agent according to claim 7, characterized in that: The intelligent agent-based video generation method also includes: recording virtual conversation content between the user agent agent and the assistant agent through an intelligent agent framework, wherein the virtual conversation content includes a function call link.

9. The video generation method based on intelligent agent according to claim 8, characterized in that: After the final video is generated, the intelligent agent-based video generation method further includes: Output the assistant agent's final answer, function directory and function call link, wherein the final answer is the generated final video, and the function directory is a function directory consisting of functions executed by the user agent in the process of generating the final video.

10. The video generation method based on intelligent agent according to claim 7, characterized in that: Before the user agent sends the user's text video requirements to the assistant agent, the intelligent agent-based video generation method also includes: obtaining the text video requirements input by the user and the selected large language model.

Images