Agent cooperation-based AIGC long video stable generation method and system

Abstract

The invention discloses an Agent cooperation-based AIGC long video stable generation method and system, and relates to the technical field of computers. Comprising the following steps: acquiring script text content input by a user; the method comprises the following steps of: pre-training an Agent of a drama group worker by adopting a preset prompt; converting the script text content input by the user into a sub-mirror script, and extracting a dialogue; extracting descriptions of roles and scenes from a script input by a user; the description of the role and the scene is converted into a standard prompt of Stable Diffusion; generating a setting graph of the role and the scene; selecting timbres conforming to role setting from a timbre library according to the role description; the timbre clone pre-training model generates a dialogue audio from the timbre and the dialogue text; generating a conversation video according to the conversation audio and the role diagram; and editing a sheet according to the split script and the dialogue video. Compared with an existing method, the role consistency and the video quality stability of AIGC long video generation are greatly improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to an agent-cooperation-based AIGC long video stable generation method and system. Background Art

[0002] There are two existing methods from video scripts to finished videos. The first is the traditional workflow of pre-planning, mid-term shooting, and post-production. It requires the collaboration of directors, producers, actors, editors and other crew members. However, the shooting and production process is very long and costly; the second is that AIGC completes the conversion of text to video, which requires generating a single scene short video and then splicing it. Multiple AI tools need to be used in the process, and the characters and scenes are inconsistent. It is very difficult to control the consistency of the characters and scenes. The lens has the problem of "going out of place" and cannot be used directly; the third is that the video currently generated by AIGC is silent, and it is difficult to generate multimodal content (audio, video) in a "one-stop" manner.

[0003] Therefore, how to improve the efficiency and quality of long video generation and reduce the cost of long video generation has become a technical problem that needs to be solved urgently. Summary of the invention

[0004] The purpose of the present invention is to provide an AIGC long video stable generation method based on Agent collaboration to address the above-mentioned technical problems existing in the prior art, so as to improve the efficiency and quality of long video generation and reduce the cost of long video generation.

[0005] Another object of the present invention is to provide an AIGC long video stable generation system based on Agent collaboration, which can realize fast, efficient and automatic generation of long videos.

[0006] In a first aspect, the present invention provides an AIGC long video stable generation method based on Agent collaboration, comprising the following steps:

[0007] 1) Get the script text content entered by the user, verify and organize its format, and report an error if it is empty;

[0008] 2) Using the preset prompt "crew member" Agent, based on the large language model, using thought chain and few prompt technology for fine-tuning, different roles call application layer tools to process the script text content;

[0009] 3) The casting agent disassembles the script and extracts descriptions of characters and scenes;

[0010] 4) The script agent uses a large language model to convert the script text into a storyboard, including scene number, shot number, and scene type information;

[0011] 5) The prompt agent converts the role and scene description into the standard prompt of the diffusion model and adds prompt words;

[0012] 6) Input the prompts of the characters and scenes into the diffusion model in the application layer to generate character and scene setting diagrams, and use the multi-view diffusion model to generate character multi-view reference diagrams and scene panorama diagrams;

[0013] 7) Select appropriate parts from the character's multi-view reference image according to the storyboard script, and merge them with the scene image to generate a storyboard;

[0014] 8) The voice agent selects the timbre that matches the role setting from the application layer timbre library according to the role description, and the timbre cloning pre-trained model generates dialogue audio based on the timbre and dialogue text;

[0015] 9) Actor Agent uses lip-motion driven technology to combine dialogue audio and character graph to generate dialogue video clips;

[0016] 10) Editing Agent Edits the storyboard and dialogue video clips into a film.

[0017] In step 1), the script text content input by the user is obtained by receiving the script text content input by the user through the user interface. This method accepts story texts and Hollywood standard format scripts. The script text content includes elements such as storyline, character dialogue, scene description, etc. The input script text content is preliminarily formatted and sorted to ensure smooth subsequent processing. If the received script text content is empty, it will stop at this step and report an error.

[0018] In step 2), the prompt engineering method has a flexible architecture. The large language model can use the self-deployed Generative Pre-training Tranformer (GPT), such as Qwen and LLaMA; it can also use Multi-LLM in a lightweight and flexible way of calling API. The preset prompt uses the Chain-of-Thought and few-shot prompt engineering technology to fine-tune the large language model. Each Agent uses a preset prompt, plays a different role, has different professional knowledge backgrounds, and can call different tools in the application layer to generate multimodal content. The agent learns how to understand and process the information in the script through the preset prompt, and through the prompt engineering technology, the agent's ability to understand the script content and the efficiency of executing subsequent tasks are improved.

[0019] In step 3), the casting agent extracts the description of roles and scenes from the script input by the user. The casting agent first disassembles the script, extracts the role elements and scene elements of the script, and stores them in json format. Furthermore, through the script and the role and scene elements, detailed information is generated for each role and scene one by one, which is equivalent to a role biography and a location scouting reference, for subsequent reference image processing.

[0020] In step 4), the script agent converts the script text or story content input by the user into a storyboard script. Specifically, the script agent uses the text understanding ability of the large language model to convert the story or script into a text storyboard and store it in the form of csv. It includes the scene, shot number, scene size, camera movement, composition guidance, scene scheduling, line information, and sound effects.

[0021] In step 5), the prompt agent converts the description of the role and scene into a standard prompt of the diffusion model. Specifically, the prompt agent converts the description of the role and scene extracted and refined by the casting agent into a format that the diffusion model can understand and process. In order to facilitate the generation of multi-view pictures, prompt words such as "white background, front, facing the camera" are added to the prompt.

[0022] In step 6), the prompts of the characters and scenes are input into the diffusion model to generate character reference images and scene reference images. Specifically, the character and scene prompts generated by the prompt agent are input into the diffusion model, and the diffusion model generates character reference images and scene reference images (512px*512px) that match the script content based on these prompts. The character reference image uses multi-view diffusion to generate images with consistent appearance features at different angles. Relative to the input image, a set of multi-view images with azimuth angles of {+0, +60, +120, +180, +240, +300} are generated. The scene reference image generates a panoramic image of the scene (1752px*584px) to facilitate subsequent further composition.

[0023] In step 7), the appropriate part is selected from the character multi-view reference picture according to the storyboard script, and the storyboard is generated by fusing with the scene picture according to the preset storyboard knowledge. For example, the character dialogue storyboard should first explain the dialogue characters, and then use the positive and negative shots to form the AB character close-up storyboard. The first storyboard character AB should be distributed in a 1:1 ratio in the picture, select character A with an azimuth angle of 60°, and select character B with an azimuth angle of 240°. In the character close-up storyboard, character A should account for 70%, select character A with an azimuth angle of 0°, and select character B with an azimuth angle of 300°. Character A is located at 1 / 3 of the width of the picture, so as to achieve reasonable arrangement and positioning of the characters.

[0024] In step 8), the voice agent selects a timbre that matches the role setting from the timbre library according to the role description. Specifically, the voice agent selects a timbre that matches the role from the timbre library according to the role description. The timbre cloning pre-training model generates dialogue audio from the timbre and dialogue text. Specifically, the timbre cloning pre-training model uses natural language processing technology and audio synthesis technology to convert the dialogue text into dialogue audio with character characteristics according to the timbre selected by the voice agent and the dialogue text extracted by the script agent.

[0025] In step 9), the actor Agent generates a video of the dialogue according to the dialogue audio and the character map. Specifically, the actor Agent generates a video of the dialogue by using lip shape driving technology, combining the dialogue audio and the character map.

[0026] In step 10), the editing agent edits the storyboard script and the dialogue video into a film. Specifically, the editing agent uses a video processing tool to splice the video clips according to the corresponding numbers of the video clips and the order of the storyboard script based on the storyboard script and the dialogue video clips generated by the actor agent to synthesize a complete long video;

[0027] Furthermore, the steps 2) to 10) are specifically as follows:

[0028] Agent collaboration can be seen as the parallelization of Chain of Thought (CoT) technology of large language models, which is divided into script, casting, prompt, sound, actor and editing agents. Each agent uses prompt engineering technology, has film and television professional knowledge and tool calling capabilities, and has multimodal information processing capabilities (Multimodal-Large Language Model), so it can effectively process multimodal information without training (training-free).

[0029] Furthermore, the steps 1) to 4) are specifically as follows:

[0030] Steps 1)-4) are designed to generate text storyboards from the script. The script (narrative story or Hollywood standard format script) is input, and the core elements (characters, locations) in the script are disassembled through the large language model and stored in the form of sequences. Using the text understanding ability of the large language model, the correspondence between the picture elements (characters, locations) and the sound elements (dialogue) is formed in the generated text storyboard, which serves as a reference for subsequent synthesis and editing.

[0031] Furthermore, the steps 5) to 7) are specifically as follows:

[0032] Steps 5)-7) are designed to form picture storyboards from the text storyboards in the previous stage. First, use the script and the disassembled elements (roles, scenes), taking into account the limit on the number of tokens generated by GPT at a time, and refine the character descriptions and scene descriptions one by one. The Prompt Agent adds generation requirements such as "white background" to the character descriptions and scene descriptions and standardizes them into prompts. The prompts are sent to the application layer diffusion model to generate character and scene reference images, and further generate multi-view images of the characters and panoramic images of the scenes. And select appropriate parts based on the storyboard expertise to synthesize into picture storyboards.

[0033] Furthermore, the steps 8)-9) are specifically as follows:

[0034] Step 8) According to the character description in step 3), the appropriate timbre is selected from the character sound library, and the character dialogue audio is further generated based on the timbre and the character dialogue in the Chinese storyboard in step 4). In step 9), the actor agent calls the application layer Wav2Lip lip shape editing method to generate a video from the character storyboard and audio.

[0035] Furthermore, the step 10) is specifically as follows:

[0036] The editing agent calls the video editing method of the application layer. According to the labels corresponding to the video clips, the video is synthesized and output into a complete long video based on the text storyboard.

[0037] In a second aspect, the present invention provides an AIGC long video stable generation system based on Agent collaboration, comprising the following modules:

[0038] The script-to-text storyboard module is used to obtain the script input by the user and convert the script into a text storyboard;

[0039] The picture storyboard generation module is used to convert the text storyboards into picture storyboards. It first disassembles the characters and scene elements in the script, and forms character and scene reference pictures after detailed descriptions, and further forms character multi-angle reference pictures and scene panorama pictures. The storyboard agent selects appropriate parts based on its professional knowledge to synthesize into picture storyboards;

[0040] The video generation module is used to generate video clips from the storyboards. It includes a dialogue audio generation module, which is used to generate dialogue audio in the video clips based on the character description and the script dialogue;

[0041] The video stitching module is used to stitch video clips into a complete video.

[0042] Furthermore, the script to text storyboard module is specifically as follows:

[0043] Get the script data and break it down into characters, scenes and other elements. Convert it into a shot-by-shot script. The shot-by-shot script is stored in the form of csv. It includes the scene, shot number, scene size, camera movement, composition guidance, scene scheduling, dialogue information, and sound effects. It also includes character information and scene information.

[0044] Furthermore, the picture storyboard generation module is specifically:

[0045] Extract characters and scenes from the script and further form descriptive information. Create a text-to-image model based on the diffusion model, and form reference images and multi-angle reference images from the descriptive information. Crop the multi-angle reference image into several parts, and select the appropriate parts to synthesize the picture storyboard.

[0046] Furthermore, the video generation module is specifically:

[0047] According to the character description, a suitable timbre is selected for the character from the timbre library, and dialogue audio is generated according to the dialogue text and timbre in the text storyboard; further, Wav2Lip is used to generate a character dialogue video clip with a shot number.

[0048] Furthermore, the video splicing module is specifically:

[0049] Based on the video clips with shot numbers and text storyboards, the video clips are assembled into a complete video.

[0050] The advantages of the present invention are:

[0051] The long video is completed by using the preset prompt to pre-train the "crew member" Agent collaboration process; the script Agent converts the script text content entered by the user into a storyboard script and extracts the dialogue from the script entered by the user; the casting Agent extracts the description of the role and scene from the script entered by the user; the prompt Agent converts the description of the role and scene into the standard prompt of the diffusion model; the prompt of the role and scene is input into the diffusion model to generate the storyboard keyframe; the sound Agent generates the timbre that conforms to the role setting according to the role description; the timbre cloning pre-training model generates the dialogue audio from the timbre and dialogue text; the actor Agent generates the dialogue video according to the dialogue audio and the character map; the editing Agent automatically edits the storyboard script and the dialogue video into a film. Applying agent collaboration to the generation of long videos can overcome the cumbersome process of multi-tool switching in the traditional process and the high cost of the actual short film shooting process, and can also effectively solve the problem of inconsistent roles and scenes in the traditional short film production process. Ultimately, the efficiency and quality of long video generation are greatly improved, and the cost of long video generation is greatly reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 The present invention is a flow chart of an agent-cooperation-based AIGC long video stable generation method and system.

[0053] Figure 2 It is a schematic diagram of an AIGC long video stable generation system based on Agent collaboration of the present invention. DETAILED DESCRIPTION

[0054] The technical solution in the embodiment of this application has the following general idea: applying agent collaboration to the generation of long videos, overcoming the cumbersome process of multi-tool switching in the traditional process and the high cost of shooting short videos, and effectively solving the problem of inconsistent roles and scenes in the traditional short film production process. Ultimately, the efficiency and quality of long video generation are greatly improved, and the cost of long video generation is greatly reduced.

[0055] Please refer to Figure 1 As shown, a preferred embodiment of an AIGC long video stable generation method and system based on Agent collaboration of the present invention includes the following steps:

[0056] Step S10, obtaining the script text content input by the user;

[0057] The script text content input by the user is received through the user interface. This method is compatible with story text and Hollywood standard format scripts. The script text content covers core elements such as storyline, character dialogue, scene description, etc. The input script text content is strictly formatted to check whether the text conforms to common script format specifications, such as whether the format of lines, scene switching, character name annotation, etc. is correct. At the same time, the text content is sorted, and extra spaces and special characters (garbled characters unrelated to the script, etc.) are removed to ensure smooth subsequent processing. If the received script text content is empty, the system will immediately feedback an error prompt and ask the user to re-enter.

[0058] Step S20: using the preset prompt "crew member" Agent, the LLM-based Agent has the ability to understand text and can also call application tools in the application layer;

[0059] The architecture of this method is highly flexible. It can use self-deployed Generative Pre-training Tranformer (GPT), such as Qwen, LLaMA, etc., to make full use of local computing resources to ensure data security and processing efficiency. Multi-LLM can also be used in a lightweight and flexible way to call APIs, with the help of the powerful computing power of the cloud to quickly respond to processing needs. The preset prompt uses Chain-of-Thought and few-shot prompt engineering technology to carefully fine-tune the large language model. Each agent is given a unique preset prompt to simulate different crew roles, such as directors, casting directors, screenwriters, etc., so as to have different professional knowledge backgrounds. For example, the prompt of the script agent will guide it to understand the overall structure of the script, the logic of plot development, and how to convert text into a visual storyboard; the prompt of the casting agent focuses on professional knowledge such as character feature analysis and image matching. These agents can call various application tools in the application layer, such as text processing tools, image generation tools, audio processing tools, etc., to generate multimodal content. Through the preset prompts, the agent can deeply learn how to accurately understand and efficiently process the information in the script, greatly improving its ability to understand the script content and the efficiency of executing subsequent tasks.

[0060] Step S30, the casting agent extracts descriptions of roles and scenes from the script input by the user;

[0061] The casting agent first disassembles the script in depth, and uses natural language processing technology to accurately identify the character elements and scene elements in the script, and stores them in json format to facilitate subsequent data calls and processing. Furthermore, using the rich information provided by the script, combined with the character and scene elements, a detailed character biography is generated for each character, including the character's personality characteristics, appearance characteristics, character background, growth experience, etc.; a comprehensive location reference is generated for each scene, such as the scene's geographical location, environmental atmosphere, spatial layout and other information. In the generation process, semantic analysis and sentiment analysis technology are used to ensure the accuracy and completeness of character and scene information, providing a solid foundation for the subsequent reference map production.

[0062] Step S40, the script Agent converts the script text content input by the user into a storyboard script;

[0063] The script agent uses the powerful text understanding ability of the Large Language Model to deeply analyze the plot development, character relationships and scene transitions of the script. The story or script is carefully converted into a professional text storyboard and stored in the form of csv to facilitate data collation and analysis. The storyboard script records in detail the scene number, shot number, scene type (such as panorama, medium shot, close-up, close-up, etc.), camera movement (push, pull, shake, move, follow, etc.), composition guidance (character position in the picture, symmetry and balance of the picture, etc.), scene scheduling (character movement, action, etc.), line information, sound effects and other key information. During the generation process, the script agent will refer to classic film and television storyboard cases and professional storyboard production specifications to ensure the professionalism and practicality of the storyboard script.

[0064] Step S50, the prompt agent converts the description of the role and the scene into a standard prompt of Stable Diffusion;

[0065] Based on the character and scene descriptions extracted and refined by the casting agent, the prompt agent uses semantic conversion and format adaptation technology to convert the descriptions into a standard format that the diffusion model can accurately understand and efficiently process. In order to generate high-quality, multi-perspective images, key prompts such as "white background, front, facing the camera" are cleverly added to the prompt. At the same time, according to the character's personality characteristics and the atmosphere requirements of the scene, targeted descriptions such as "determined expression" and "mysterious light and shadow effects" are added to guide the diffusion model to generate images that meet expectations.

[0066] Step S60: input the prompt of the character and the scene into Stable Diffusion in the application layer to generate the setting map of the character and the scene; and further use the multi-view diffusion model to generate a multi-view reference map of the character and a panoramic map of the scene.

[0067] The character and scene prompts carefully generated by the prompt agent are input into the StableDiffusion model in the application layer. Based on these prompts, the model uses deep learning algorithms and image generation technology to generate character reference images and scene reference images (512px * 512px) that are highly consistent with the script content. For the character reference image, the advanced multi-view diffusion technology is used to generate pictures with consistent appearance features at different angles. Relative to the input picture, a set of multi-view pictures with azimuth angles of {+0, +60, +120, +180, +240, +300} are generated to fully display the appearance of the character. The scene reference image generates a panoramic view of the scene (1752px * 584px) through an optimized diffusion model, providing a broad and rich scene foundation for subsequent further composition. During the generation process, the parameters of the model are finely adjusted, and the generation strategy of the model is optimized according to different script styles and requirements, such as science fiction, ancient costumes, modern, etc., to ensure that the quality and style of the generated image are consistent with the overall setting of the script.

[0068] Step S70: Select appropriate parts from the character multi-view reference image according to the storyboard script, and fuse them with the scene image to generate a storyboard.

[0069] According to the preset storyboard knowledge and professional composition principles, the appropriate parts are accurately selected from the character multi-view reference map, and cleverly integrated with the scene picture to generate high-quality storyboards. For example, in the character dialogue storyboard, first follow the conventional techniques of film and television shooting to explain the dialogue characters, and then use the method of positive and negative shots to form AB character close-up storyboards. In the first explanation storyboard, the proportion of characters AB in the picture should be 1:1. Select character A with an azimuth of 60° and character B with an azimuth of 240° to show the relative position and communication status of the two parties in the dialogue. In the character close-up storyboard, character A should account for 70%, select character A with an azimuth of 0°, select character B with an azimuth of 300°, and character A is located at 1 / 3 of the width of the picture. Through this precise selection and layout, the reasonable arrangement and precise positioning of the characters in the picture can be achieved, and the expressiveness and narrative of the storyboard can be enhanced. During the fusion process, image fusion algorithms are used to ensure that elements such as light, shadow, and color are coordinated between the characters and the scene to create a realistic visual effect.

[0070] Step S80: The sound agent selects a timbre that matches the role setting from the application layer timbre library according to the role description; the timbre cloning pre-trained model generates a dialogue audio from the timbre and dialogue text;

[0071] Based on the detailed description of the character, including personality traits, age, identity and other information, the voice agent intelligently screens from the rich timbre library at the application layer and selects a timbre that is highly consistent with the character. The timbre cloning pre-training model uses advanced natural language processing technology and audio synthesis technology to perform speech synthesis on the text based on the timbre selected by the voice agent and the dialogue text extracted by the script agent. During the synthesis process, the model fully considers factors such as the tone, speed, and emotional expression of the voice, and dynamically adjusts the voice parameters according to the plot of the dialogue and the emotional changes of the character, so that the generated dialogue audio has distinct character characteristics. The timbre cloning pre-training model is trained based on a large amount of voice data, covering a variety of language styles, emotional types and timbre characteristics to ensure that high-quality, natural and smooth dialogue audio can be generated.

[0072] Step S90, the actor Agent generates a video clip of the dialogue according to the dialogue audio and the character map;

[0073] Actor Agent uses advanced lip-shaped driving technology, combined with dialogue audio and character graphs, to generate realistic dialogue video clips. Through voice analysis of the dialogue audio, the phonemes, rhythm and other information of the voice are accurately extracted, and then the lip movements in the character graph are driven based on this information to achieve accurate synchronization of lip shape and voice. During the generation process, the lip shape movements are personalized to take into account the speaking habits and facial expressions of different characters, making the character performance in the video clip more vivid and natural. At the same time, using image rendering and animation technology, appropriate facial expressions and body movements are added to the characters to enhance the expressiveness and appeal of the video.

[0074] Step S100: The editing agent edits the storyboard and the dialogue video clips into a film.

[0075] The editing agent uses professional video processing tools to accurately splice the video clips in the order of the storyboard script according to the detailed planning of the storyboard script and the dialogue video clips generated by the actor agent. During the splicing process, according to the corresponding labels of the video clips, ensure the correct position of each clip on the timeline, and consider the transition effects between shots, such as fade in and fade out, flash cut, rotation, etc., so that the video transition is natural and smooth. Control the overall rhythm of the video, adjust the video playback speed and editing rhythm according to the tension of the plot and the emotional ups and downs, and enhance the viewing and attractiveness of the video. Finally, the spliced ​​video is optimized and adjusted as a whole, including post-processing such as color correction and audio mixing, to synthesize a complete high-quality long video.

[0076] like Figure 2 , a preferred embodiment of the AIGC long video stable generation system based on Agent collaboration of the present invention comprises the following modules in sequence: a script to text storyboard module, a picture storyboard generation module, a dialogue audio generation module, a video clip generation module, and a video splicing module;

[0077] The script-to-text storyboard module: As the starting module of the entire system, it is responsible for processing the script text input by the user and breaking the script into elements such as characters and scenes. Convert it into a storyboard script. The storyboard script is stored in the form of csv. It includes scenes, shot numbers, scene sizes, camera movements, composition guidance, scene scheduling, dialogue information, and sound effects. It also includes character information and scene information; its core is to carefully break down the various elements in the script and present them in a clear and organized manner, providing a basic framework for the creation of long videos. In addition, the module performs a preliminary logic check on the generated storyboard script to ensure the coherence between the storyboards and the rationality of the plot development. For possible logical loopholes, the script structure is optimized through intelligent prompts or automatic adjustments to ensure the fluency of subsequent video creation.

[0078] The picture storyboard generation module: based on the storyboard script output by the script to text storyboard module, creates a corresponding visual image for each storyboard, creates a text-to-image model based on the diffusion model, and forms reference images and multi-angle reference images from the description information. The multi-angle reference image is cut into several parts, and the appropriate parts are selected to synthesize the picture storyboard. This provides a basis for the visual presentation of the video and ensures that the image matches the script content.

[0079] The dialogue audio generation module selects a suitable timbre for the character from the timbre library according to the character description, and generates dialogue audio according to the dialogue text and timbre in the text storyboard; gives the character in the video a voice with personality and emotion, making the character's language expression more vivid and more in line with the character image and plot atmosphere.

[0080] The video clip generation module generates character dialogue video clips with mirror numbers based on dialogue audio and character images through Wav2Lip, organically combining the character's language expression and action expression to make the character vivid in the video.

[0081] The video splicing module: splices the video clips into a complete long video according to the video clips with shot numbers and text storyboards, and can perform post-processing on the video clips to make the entire video coherent and smooth.

[0082] The modules transmit and share information through a unified data interface. The output data of each module is stored in a standardized format to facilitate the call and processing of other modules. The storyboard script output by the script to text storyboard module is stored in CSV format, and other modules can easily read the information in it; the image data generated by the picture storyboard generation module adopts a universal image format and contains corresponding metadata so that subsequent modules can operate on it.

[0083] During the operation of the system, error monitoring and handling mechanisms can be set up. Possible input errors, model operation errors, data processing errors, etc. can be monitored in real time. When an error occurs, the system will provide the user with clear error information based on the error type, and try to automatically repair or provide corresponding solutions. If there is a quality problem with the image generated by the table Diffusion type, the system will automatically adjust the model parameters or use an alternative image generation algorithm; if there is a problem with audio synthesis, try to use different audio synthesis parameters or reselect the timbre.

[0084] The following is an example of making a long animation video about "Little Red Riding Hood" to illustrate the scheme of the present invention. The script is divided into several segments: Little Red Riding Hood bids farewell to her mother; the big bad wolf eats her grandmother; Little Red Riding Hood goes to her grandmother's house and is eventually eaten by the big bad wolf; the hunter arrests the big bad wolf and saves Little Red Riding Hood and her grandmother.

[0085] Step 1: Get the script text content entered by the user:

[0086] The user inputs the script text about "Little Red Riding Hood" through the user interface. The script contains detailed storylines, character dialogues, scene descriptions and other elements. The system performs preliminary format verification and sorting on the input script to ensure smooth subsequent processing.

[0087] Step 2: Pre-train the "Crew" Agent using the preset prompt

[0088] The system uses preset prompts to pre-train the script agent, casting agent, prompt agent, sound agent, actor agent, and editing agent. These agents learn how to understand and process the information in the script through preset prompts, improving their ability to understand the script content and the efficiency of executing subsequent tasks. Tools in the application layer include: Stable diffusion and multi-view diffusion, character timbre library, timbre cloning model, and Wav2Lip lip shape driver tool.

[0089] Step 3: The casting agent extracts the core elements from the script text input by the user: roles, scenes; and describes them one by one. For example: From "Little Red Riding Hood", the characters are extracted and numbered: Little Red Riding Hood 0, Mom 1, Grandma 2, Big Bad Wolf 3, Hunter 4. The scenes are extracted: Little Red Riding Hood's home 0, Forest 1, Grandma's home 2. Next, the descriptions are refined one by one:

[0090]

[0091] Step 4: The script agent uses the large language model to convert the script into text storyboards.

[0092] For example, (Scene 1) Little Red Riding Hood says goodbye to her mother.

[0093]

[0094] And disassembled into storyboard text script

[0095]

[0096] Step 5: The prompt agent converts the description of the role and scene into a standard prompt for Stable Diffusion.

[0097]

[0098] Step 6. Input the prompt of the character and scene into Stable Diffusion in the application layer to generate the setting map of the character and scene (512px*512px). And further use the multi-view diffusion model to generate the multi-view reference map of the character (azimuth angle is {+0, +60, +120, +180, +240, +300}) and the panoramic map of the scene (1080px*584px).

[0099] Step 7: Select the appropriate part from the character's multi-view reference picture according to the storyboard script, and merge it with the scene picture to generate a storyboard.

[0100]

[0101] Step 8: The voice agent selects a timbre that matches the role setting from the application layer timbre library based on the role description; for example, a clear and pleasant child female voice is selected for Little Red Riding Hood; a young female voice is selected for the mother. The timbre cloning pre-trained model generates dialogue audio from the timbre and dialogue text; for example: Audio 0 is generated for mirror number 2: "Goodbye, Mom!" Audio 1 is generated for mirror number 2: "Be careful on the road."

[0102] Step 9: The actor agent generates a video clip of the dialogue based on the dialogue audio and the character graph; uses audio 0 on storyboard image 2 for lip matching. Generates video 1. Uses audio 1 on storyboard image 3 for lip matching. Generates video 2.

[0103] Step 10: The editing agent edits the storyboard and the dialogue video clips into a film, i.e. rents the scenes and shot numbers in order.

[0104] The present invention adopts multi-agent division of labor and cooperation to modularize the long video generation process to avoid tedious manual operations. The large language model combined with prompt engineering technology can quickly process script information, accelerate the generation of storyboard scripts, and shorten the production cycle. Reduce dependence on manpower and reduce labor costs; no actual shooting is required, saving expenses such as venues, props, equipment and actor remuneration. The information extracted from the script in the present invention runs through the entire process to ensure the consistency of roles and scenes; advanced technology is used to ensure the high quality and stability of images, audio, and video, and optimize transitions and post-processing during editing. The present invention has a flexible architecture and can be deployed locally or in the cloud; it can adapt to different script styles and languages, and can expand functional modules or update algorithms. The present invention facilitates users to quickly convert creativity into long videos. It effectively solves the problem of inconsistent roles and scenes in the production process of traditional short films, improves the efficiency and quality of long video generation, and reduces the cost of long video generation.

[0105] Although the specific implementation modes of the present invention are described above, those skilled in the art should understand that the specific implementation modes described are only illustrative and are not intended to limit the scope of the present invention. Equivalent modifications and changes made by those skilled in the art in accordance with the spirit of the present invention should be included in the scope of protection of the claims of the present invention.

Claims

1. A stable generation method of long AIGC videos based on Agent collaboration, characterized in that The steps include: 1) Obtain the script text content input by the user, verify and organize its format, and report an error if it is empty; the script text content includes the storyline, character dialogue, and scene description; 2) Using the preset prompt "crew member" Agent, based on the large language model, using thought chain and few prompt technology for fine-tuning, different roles call application layer tools to process the script text content; 3) The casting agent disassembles the script and extracts descriptions of characters and scenes; 4) Script Agent uses a large language model to convert the script text content into a storyboard, including scene number, shot number, shot size, camera movement, composition guidance, scene scheduling, dialogue information, and sound effects; 5) The prompt agent converts the role and scene description into the standard prompt of the diffusion model and adds prompt words; 6) Input the prompts of the characters and scenes into the diffusion model in the application layer to generate character and scene setting diagrams, and use the multi-view diffusion model to generate character multi-view reference diagrams and scene panorama diagrams; 7) Select appropriate parts from the character's multi-view reference image according to the storyboard script, and merge them with the scene image to generate a storyboard; 8) The voice agent selects the timbre that matches the role setting from the application layer timbre library according to the role description, and the timbre cloning pre-trained model generates dialogue audio based on the timbre and dialogue text; 9) Actor Agent uses lip-motion driven technology to combine dialogue audio and character graph to generate dialogue video clips; 10) Editing Agent Edits the storyboard and dialogue video clips into a film.

2. The method for generating stable long videos using AIGC based on Agent collaboration as claimed in claim 1, characterized in that: In step 2), the large language model is a self-deployed GPT, using Qwen or LLaMA; Or by calling the Multi-LLM of the API; the preset prompt uses thought chaining and few-time prompt engineering technology to fine-tune the large language model. Each Agent uses a preset prompt, plays a different role, has different professional knowledge backgrounds, and calls different tools in the application layer to generate multimodal content; the Agent learns how to understand and process the information in the script through the preset prompt, and through prompt engineering technology, improves the Agent's ability to understand the script content and the efficiency of executing subsequent tasks.

3. The method for generating stable long videos using AIGC based on agent collaboration as claimed in claim 1, characterized in that: In step 3), the description of the characters and scenes includes character biographies and location scouting references, covering the character's personality, appearance, background information, and the location, environment, atmosphere, and spatial layout of the scene.

4. The method for generating stable long videos using AIGC based on agent collaboration as claimed in claim 1, characterized in that: In step 5), the prompt agent converts the role and scene description into a standard prompt of the diffusion model and adds prompt words. Specifically, the prompt agent converts the description into a format that can be understood and processed by the diffusion model based on the role and scene description extracted and refined by the casting agent; and in order to facilitate the generation of multi-perspective images, prompt words such as "white background, front, facing the camera" are added to the prompt.

5. The method for generating stable long videos using AIGC based on Agent collaboration as claimed in claim 1, characterized in that: In step 6), the character multi-view reference map and the scene panoramic map are generated, specifically: the character and scene prompts generated by the prompt agent are input into the diffusion model, and the diffusion model generates the character reference map and the scene reference map that are consistent with the script content according to these prompts; the character reference map uses multi-view diffusion to generate pictures with consistent appearance features at different angles, and relative to the input picture, a set of multi-view pictures with azimuth angles of {+0, +60, +120, +180, +240, +300} are generated; the scene reference map generates a panoramic view of the scene to facilitate subsequent further composition.

6. The method for generating stable long videos using AIGC based on Agent collaboration as claimed in claim 1, characterized in that: In step 7), when generating the storyboard, for the character dialogue storyboard, first explain the dialogue characters, and use the method of positive and negative shots to form AB character close-up storyboards. In the first storyboard, the ratio of character AB is 1:1, and character A with an azimuth angle of 60° and character B with an azimuth angle of 240° are selected respectively; in the character close-up storyboard, character A accounts for 70%, and character A with an azimuth angle of 0° and character B with an azimuth angle of 300° are selected, and character A is located at 1 / 3 of the width of the picture to achieve reasonable arrangement and positioning of the characters.

7. The method for generating stable long videos using AIGC based on Agent collaboration as claimed in claim 1, characterized in that: In step 9), the actor agent adds facial expressions and body movements that match the lines and emotions to the character when generating dialogue video clips.

8. The method for generating stable long videos using AIGC based on Agent collaboration as claimed in claim 1, characterized in that: In step 10), the editing agent edits the storyboard script and the dialogue video clips into a film. The editing agent uses a video processing tool to splice the video clips according to the corresponding numbers of the video clips and the order of the storyboard script based on the storyboard script and the dialogue video clips generated by the actor agent to synthesize a complete long video; when splicing the video, the editing agent uses a variety of transition effects, including fade in and fade out, flash cut or rotation, and performs color correction and audio mixing on the video.

9. An AIGC long video stable generation system based on Agent collaboration, characterized in that: Includes the following modules: The script-to-text storyboard module is used to obtain the script input by the user and convert the script into a text storyboard containing scene number, shot number, scene size, camera movement, composition guidance, scene scheduling, dialogue information and sound effect information; A picture storyboard generation module is used to convert the characters and scene elements in the text storyboard into multi-angle reference pictures of the characters and panoramic pictures of the scene, and synthesize the picture storyboard according to the storyboard knowledge; A video generation module is used to generate video clips from the storyboards; the video generation module includes a dialogue audio generation module, which is used to generate dialogue audio in the video clips based on the character description and the script dialogue; The video splicing module is used to splice video clips with shot numbers into a complete video according to text shots.

10. The AIGC long video stable generation system based on Agent collaboration as claimed in claim 9, characterized in that: Each module exchanges data through a data interface, and data transmission between modules follows a unified standard format.

Images