Multimedia animation generation method

Abstract

The invention relates to a multimedia animation generation method, which comprises the steps of pre-production and blueprint design, static visual asset generation, dynamic and audio generation, synthesis and later stage, the multimedia animation generation method adopts AI to carry out auxiliary creation, and the creation can be completed by cooperating with a consumer-level GPU and a part of open source tools. The capital and professional skill threshold is greatly reduced, various designs of the same role, various styles of the same scene and the like can be quickly and efficiently generated, and diversification of creation ecology is promoted.

Description

Technical Field

[0001] This invention relates to a method for generating multimedia animation. Background Technology

[0002] Traditional animation production requires a large professional team and expensive hardware and software investment, resulting in high costs. Furthermore, due to cost control, traditional animation production cannot amplify creative exploration capabilities, and there are significant limitations in the design requirements for characters, the experimentation with scene styles, and the exploration of narrative branches. In particular, small teams cannot effectively create large-scale animations. Summary of the Invention

[0003] This invention provides a method for generating multimedia animation.

[0004] The technical solution adopted by this invention to solve its technical problem is: a multimedia animation generation method, comprising the following steps:

[0005] a. Pre-production and blueprint design: Transforming vague ideas into executable, structured sets of AI-generated instructions, including story and script generation and character and scene setting;

[0006] b. Static visual asset generation: Based on the design document in step a, generate all the required static images, including core character designs and scene and background designs;

[0007] c. Animation and Audio Generation: To bring static images to life and generate voiceovers and sound effects, including animation generation and audio generation;

[0008] d. Compositing and Post-Production: Combining all elements into the final video, including video editing, compositing, and final output.

[0009] As a preferred option, in step a, story and script generation uses a large language model. By inputting keywords such as story outline, theme, and style, and through multiple rounds of dialogue, the LLM generates a detailed story outline and episode scripts, including complete character dialogues and scene descriptions.

[0010] As a preferred option, in step a, character and scene setting, LLM is used to generate detailed profiles for each main character, including name, age, personality, physical characteristics, and clothing details, to generate character profiles; the script is broken down into multiple independent shots, and detailed text descriptions are generated for each shot, including character actions, expressions, camera angles, scene layout, etc.

[0011] As a preferred approach, in step b, the core character design uses a text-based image model. The character's appearance description is input from their profile, and a batch of images is generated. One image is selected, and a LoRA model for that character is trained to lock onto that image. All subsequent generation uses this LoRA model. The OpenPose preprocessor is used to precisely control the character's pose, thereby creating a character expression and pose library. Scene and background generation uses a text-based image model. The scene portion of the storyboard description is input, and the image-based image function is used to first draw a simple layout sketch, then refine and color it to achieve better composition control.

[0012] Preferably, the specific steps for animation generation in step c are as follows:

[0013] C1, 2D sequence generation: Using the AnimateDiff model, a short video sequence is generated from a still image by inputting a cue word describing the camera movement;

[0014] C2. Motion Control: Combined with ControlNet, it guides the movement trajectory of the character or camera by controlling changes in the skeleton or depth map;

[0015] C3. Lip-sync: Using SadTalker and Wav2Lip tools, the generated character voice is matched with the character's face to automatically generate lip-sync animation.

[0016] Preferably, step c, the audio generation step, includes:

[0017] D1. Character voice acting: Using emotional TTS tools, assign a unique voice to each character and convert script dialogue into audio files;

[0018] D2. Background Music and Sound Effects: Using the AIVA music generation tool, BGM is generated from prompt words, and ambient sound effects are generated from the sound effects library.

[0019] Preferably, step d includes the following specific steps for video editing, compositing, and output:

[0020] E1. In professional software, splice the generated animation sequence according to the storyboard order;

[0021] E2. Refine the video clips after lip-syncing;

[0022] E3. Import all audio files for precise audio-visual synchronization;

[0023] E4, add subtitles, visual effects, and color correction, then render and export the final video file.

[0024] The beneficial effects of this invention are that the multimedia animation generation method uses AI for assisted creation, and can complete the creation with the help of consumer-grade GPUs and some open-source tools, which greatly reduces the financial and professional skill thresholds, and can quickly and efficiently generate multiple designs of the same character, multiple styles of the same scene, etc., thus promoting the diversification of the creative ecosystem. Attached Figure Description

[0025] Figure 1 This is a flowchart illustrating a multimedia animation generation method provided in an embodiment of the present invention. Detailed Implementation

[0026] like Figure 1 As shown in the figure, an embodiment of the present invention provides a multimedia animation generation method, which includes the following steps:

[0027] a. Pre-production and blueprint design: Transforming vague ideas into executable, structured sets of AI-generated instructions, including story and script generation and character and scene setting;

[0028] The technical essence of this stage is actually the brain of the project. Its output is not the final product, but a sophisticated set of instructions that drives all subsequent AI work.

[0029] b. Static visual asset generation: Based on the design document in step a, generate all the required static images, including core character designs and scene and background designs;

[0030] The technical essence of this step is actually the project's art department, which is responsible for creating all the visual elements.

[0031] c. Animation and Audio Generation: To bring static images to life and generate voiceovers and sound effects, including animation generation and audio generation;

[0032] This stage essentially involves the project's animation and audio departments bringing static assets to life.

[0033] d. Compositing and Post-Production: Combining all elements into the final video, including video editing, compositing, and final output.

[0034] This stage is actually the project's final assembly workshop, where all components are assembled into the final product.

[0035] As a preferred option, in step a, story and script generation uses a large language model. By inputting keywords such as story outline, theme, and style, and through multiple rounds of dialogue, the LLM generates a detailed story outline and episode scripts, including complete character dialogues and scene descriptions.

[0036] Large Language Model (LLM) can not only expand creativity, but also generate structured scripts containing dialogue and scene descriptions, providing context for subsequent generation. Furthermore, the quality of the cue words directly determines the relevance and quality of the generated content.

[0037] As a preferred option, in step a, character and scene setting, LLM is used to generate detailed profiles for each main character, including name, age, personality, physical characteristics, and clothing details, to generate character profiles; the script is broken down into multiple independent shots, and detailed text descriptions are generated for each shot, including character actions, expressions, camera angles, scene layout, etc.

[0038] This is a crucial shift from storytelling to visual representation. The character profiles generated by LLM serve as the requirements document for visual generation, while the storyboard descriptions are the shooting scripts for each shot, detailing camera angles and scene layouts. This provides the foundation for accurate prompts in the next step of text-to-image modeling, avoiding randomness and bias in the generation process.

[0039] As a preferred approach, in step b, the core character design uses a text-based image model. The character's appearance description is input from their profile, and a batch of images is generated. One image is selected, and a LoRA model for that character is trained to lock onto that image. All subsequent generation uses this LoRA model. The OpenPose preprocessor is used to precisely control the character's pose, thereby creating a character expression and pose library. Scene and background generation uses a text-based image model. The scene portion of the storyboard description is input, and the image-based image function is used to first draw a simple layout sketch, then refine and color it to achieve better composition control.

[0040] The application of the LoRA model is the core technology to ensure character consistency. Without LoRA, the generated character image would be inconsistent each time. Training a LoRA model specifically for a character is equivalent to creating a standard ID photo for the AI, fundamentally solving the character drift problem in AI animation.

[0041] Preferably, the specific steps for animation generation in step c are as follows:

[0042] C1, 2D sequence generation: Using the AnimateDiff model, a short video sequence is generated from a still image by inputting a cue word describing the camera movement;

[0043] C2. Motion Control: Combined with ControlNet, it guides the movement trajectory of the character or camera by controlling changes in the skeleton or depth map;

[0044] C3. Lip-sync: Using SadTalker and Wav2Lip tools, the generated character voice is matched with the character's face to automatically generate lip-sync animation.

[0045] The application of ControlNet is key to ensuring motion controllability. It allows creators to direct characters to pose in specific ways, thereby efficiently generating the same character's poses in different scenes and building a reusable asset library.

[0046] ControlNet is used here to guide character walking or camera movement by controlling changes in the skeleton or depth map, making the movement more in line with the director's intentions.

[0047] The generation of scenes and backgrounds is the image-to-image function. This preferred solution emphasizes control. Simple text-to-image generation may result in a background with a chaotic composition. By drawing a simple layout sketch, the creator takes control of the composition, while the AI ​​performs advanced coloring and detail rendering, achieving human-computer collaborative creation.

[0048] The AnimateDiff model is currently the mainstream model for AI video, which can transform the potential of static images into dynamic sequences.

[0049] Using SadTalker and Wav2Lip tools can enhance the professionalism of your work by automatically synchronizing audio and video streams, eliminating the tedious manual work of audio-visual matching and greatly improving work efficiency.

[0050] Preferably, step c, the audio generation step, includes:

[0051] D1. Character voice acting: Using emotional TTS tools, assign a unique voice to each character and convert script dialogue into audio files;

[0052] D2. Background Music and Sound Effects: Using the AIVA music generation tool, BGM is generated from prompt words, and ambient sound effects are generated from the sound effects library.

[0053] Using emotional TTS tools can make characters more vivid and improve the presentation.

[0054] The combination of AI music generation and sound effects library creates a complete auditory environment.

[0055] Preferably, step d includes the following specific steps for video editing, compositing, and output:

[0056] E1. In professional software, splice the generated animation sequence according to the storyboard order;

[0057] E2. Refine the video clips after lip-syncing;

[0058] E3. Import all audio files for precise audio-visual synchronization;

[0059] E4, add subtitles, visual effects, and color correction, then render and export the final video file.

[0060] In this step, manual retouching is performed because AI-generated footage is usually rough and fragmented. The sequence splicing here is equivalent to the rough cut in traditional editing. The retouching in step E2 is a key step to deal with the defects generated by AI. It often involves manually fixing lip-syncing flaws or animation jumps. Step E3 is to ensure the unity of auditory and visual experience, while step E4 is to improve the cinematic feel of the work and enhance the overall quality of the work.

[0061] Compared with existing technologies, this multimedia animation generation method uses AI to assist in creation, and can complete the creation with the help of consumer-grade GPUs and some open-source tools, which greatly reduces the financial and professional skill thresholds. It can also quickly and efficiently generate multiple designs of the same character and multiple styles of the same scene, thus promoting the diversification of the creative ecosystem.

Claims

1. A method for generating multimedia animation, characterized in that, Includes the following steps: a. Pre-production and blueprint design: Transforming vague ideas into executable, structured sets of AI-generated instructions, including story and script generation and character and scene setting; b. Static visual asset generation: Based on the design document in step a, generate all the required static images, including core character designs and scene and background designs; c. Animation and Audio Generation: To bring static images to life and generate voiceovers and sound effects, including animation generation and audio generation; d. Compositing and Post-Production: Combining all elements into the final video, including video editing, compositing, and final output.

2. The multimedia animation generation method as described in claim 1, characterized in that, In step a, story and script generation uses a large language model. By inputting keywords such as story outline, theme, and style, and through multiple rounds of dialogue, the LLM generates a detailed story outline and episode scripts, including complete character dialogues and scene descriptions.

3. The multimedia animation generation method as described in claim 1, characterized in that, In step a, character and scene settings are created. LLM is used to generate detailed profiles for each main character, including name, age, personality, physical characteristics, and clothing details, thus generating character profiles. The script is broken down into multiple independent shots, and detailed text descriptions are generated for each shot, including character actions, expressions, camera angles, scene layout, etc.

4. The multimedia animation generation method as described in claim 3, characterized in that, In step b, the core character design uses a text-based image model. The character's appearance description is input, generating a batch of images. One image is selected, and a LoRA model for that character is trained to lock onto that image. All subsequent generation uses this LoRA model. The OpenPose preprocessor is used to precisely control the generated character's pose, thus creating a character expression and pose library. Scene and background generation uses a text-based image model. The scene portion of the storyboard description is input, and the image-based image function is used to first draw a simple layout sketch, then refine and color it to achieve better composition control.

5. The multimedia animation generation method as described in claim 1, characterized in that, In step c, the specific steps for generating the animation are as follows: C1, 2D sequence generation: Using the AnimateDiff model, a short video sequence is generated from a still image by inputting a cue word describing the camera movement; C2. Motion Control: Combined with ControlNet, it guides the movement trajectory of the character or camera by controlling changes in the skeleton or depth map; C3. Lip-sync: Using SadTalker and Wav2Lip tools, the generated character voice is matched with the character's face to automatically generate lip-sync animation.

6. The multimedia animation generation method as described in claim 1, characterized in that, In step c, the audio generation steps include: D1. Character voice acting: Using emotional TTS tools, assign a unique voice to each character and convert script dialogue into audio files; D2. Background Music and Sound Effects: Using the AIVA music generation tool, BGM is generated from prompt words, and ambient sound effects are generated from the sound effects library.

7. The multimedia animation generation method as described in claim 1, characterized in that, In step d, the specific steps for video editing, compositing, and output include: E1. In professional software, splice the generated animation sequence according to the storyboard order; E2. Refine the video clips after lip-syncing; E3. Import all audio files for precise audio-visual synchronization; E4, add subtitles, visual effects, and color correction, then render and export the final video file.

Images