Video generation methods, apparatus, electronic devices and machine-readable media

By employing a segmented debugging mechanism and a multimodal large model, the problem of low video optimization efficiency in animation generation was solved, achieving automation and rapid optimization of the video generation workflow.

CN123123669APending Publication Date: 2026-11-17UC MOBILE CHINA CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611241356.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-08-14
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing animation generation solutions require regenerating the entire video when there are video quality issues, resulting in low optimization efficiency.

Method used

The workflow is generated by acquiring target text and video, recording task output data, and re-executing specified text segments and tasks when receiving segmented debugging instructions. Segmented debugging is achieved using multimodal large models and FFmpeg tools to generate target videos.

Benefits of technology

It improved the efficiency of video generation optimization and reduced production time from more than 5 days to within 8 hours, realizing the automation of the video generation workflow and the segmented debugging of some tasks.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123123669A_ABST
    Figure CN123123669A_ABST
Patent Text Reader

Abstract

This application provides a video generation method, apparatus, electronic device, and machine-readable medium, relating to the field of artificial intelligence technology. The method includes: acquiring target text and a video generation workflow, the video generation workflow including multiple tasks with dependencies; executing the video generation workflow for the target text and recording the output data of each task for each text segment in the target text; upon receiving a segmented debugging instruction, determining the re-execution text segment and the re-execution task; the re-execution text segment includes the text segment in the target text corresponding to the target content indicated by the segmented debugging instruction; the re-execution task includes the target task indicated by the segmented debugging instruction and downstream tasks; based on the output data of the directly dependent tasks of the re-execution task and the re-execution text segment, re-executing the re-execution task, and obtaining the target video upon completion of the video generation workflow. This application improves video optimization efficiency.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a video generation method, apparatus, electronic device, and machine-readable medium. Background Technology

[0002] With the rapid development of video applications, AI (Artificial Intelligence) comics have emerged. AI comics are comic videos intelligently generated based on AI technology and have the advantage of low cost.

[0003] However, current animation generation solutions can only regenerate the entire video when quality issues arise, resulting in low video optimization efficiency. Summary of the Invention

[0004] This application provides a video generation method to solve the technical problem of low video optimization efficiency.

[0005] Accordingly, embodiments of this application also provide a video generation apparatus, an electronic device, and a machine-readable medium to ensure the implementation and application of the above methods.

[0006] To address the aforementioned problems, this application discloses a video generation method, which includes: The process involves obtaining the target text and video generation workflow. The video generation workflow includes multiple tasks with dependencies, and the target text includes at least one text segment. The video generation workflow is executed on the target text, and the output data of the tasks for at least one text segment is recorded. Upon receiving a segmented debugging instruction, the process determines the re-execution text segment and the re-execution task. The re-execution text segment includes the text segment corresponding to the target content indicated by the segmented debugging instruction within the target text. The re-execution task includes the target task indicated by the segmented debugging instruction and its downstream tasks in the video generation workflow. Based on the output data of the directly dependent tasks of the re-execution task and the re-execution text segment, the re-execution task is re-executed, and the target video corresponding to the target text is obtained upon completion of the video generation workflow.

[0007] This application discloses a video generation apparatus, including: The acquisition module is used to acquire the target text and video generation workflow. The video generation workflow includes multiple tasks with dependencies, and the target text includes at least one text segment.

[0008] The execution module is used to execute a video generation workflow for the target text and record the output data of the task for at least one text segment.

[0009] The determination module is used to determine the re-execution text segment and the re-execution task upon receiving a segmented debugging instruction. The re-execution text segment includes the text segment in the target text corresponding to the target content indicated by the segmented debugging instruction. The re-execution task includes the target task indicated by the segmented debugging instruction and the downstream task of the target task in the video generation workflow.

[0010] The first execution module is used to re-execute the re-execution task based on the output data of the direct dependent tasks of the re-execution task and the re-execution text segment, and obtain the target video corresponding to the target text when the video generation workflow is completed.

[0011] This application also discloses an electronic device, including: a processor; and a memory storing executable code thereon, which, when executed, causes the processor to perform one or more methods as described in the embodiments of this application.

[0012] This application also discloses one or more machine-readable media storing executable code thereon, which, when executed, causes a processor to perform one or more methods as described in this application.

[0013] Compared with related technologies, the embodiments of this application include the following advantages: obtaining target text and video generation workflow, the video generation workflow including multiple tasks with dependencies; executing the video generation workflow for the target text and recording the output data of the tasks for at least one text segment; upon receiving a segmented debugging instruction, determining the re-execution text segment and the re-execution task; the re-execution text segment includes the text segment in the target text corresponding to the target content indicated by the segmented debugging instruction; the re-execution task includes the target task indicated by the segmented debugging instruction and the downstream task of the target task in the video generation workflow; based on the output data of the directly dependent tasks of the re-execution task and the re-execution text segment, re-execute the re-execution task, and obtain the target video corresponding to the target text when the video generation workflow is completed. The embodiments of this application have the following beneficial effects: Firstly, by recording the output data of each task during the execution of the video generation workflow, the re-execution of some tasks and some text segments is supported, realizing segmented debugging of the video generation workflow. Compared with re-execution of the entire workflow, this can effectively improve the efficiency of video optimization. Secondly, by generating videos in a fully automated manner through the video generation workflow, no human intervention is required during the execution of the video generation workflow, which helps to improve the efficiency of video generation. Compared with the prior art, which requires more than 5 days to produce a comic series, the embodiments of this application can reduce the production time to less than 8 hours. Attached Figure Description

[0014] Figure 1 This is a schematic diagram of a hardware and software layered framework provided in an embodiment of this application; Figure 2 The embodiments provided in this application are based on Figure 1 A schematic diagram illustrating the configuration and execution process of a video generation workflow; Figure 3 This is a flowchart illustrating the steps of a video generation method provided in an embodiment of this application; Figure 4 This is a schematic diagram of the DAG structure of a video generation workflow provided in an embodiment of this application; Figure 5 This is a schematic diagram illustrating the execution steps of a video generation workflow provided in an embodiment of this application; Figure 6 This is a schematic diagram illustrating the changes in the queue to be executed provided in an embodiment of this application; Figure 7 This is a schematic diagram illustrating the execution steps of another video generation workflow provided in this application embodiment; Figure 8 This is a schematic diagram of the structure of a video generation device provided in an embodiment of this application; Figure 9 This is a schematic diagram of another video generation device provided in an embodiment of this application; Figure 10 This is a schematic diagram of the structure of a device provided in an embodiment of this application. Detailed Implementation

[0015] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0016] To enable those skilled in the art to better understand this application, the concepts involved in this application are explained below: Workflow Orchestration Engine: This refers to a task scheduling system that defines task dependencies using a DAG (Directed Acyclic Graph) and supports complex scheduling modes such as parallel, serial, and cyclic retries to automate the production chain of AI animation series. In this embodiment, the workflow orchestration engine includes: a workflow platform layer, a workflow orchestration service layer, a workflow execution service layer, an engine layer, and a task interface layer.

[0017] DAG: A graph structure used to describe the dependencies between tasks. In this embodiment, nodes in a DAG represent tasks, and edges in a DAG represent dependencies between tasks, ensuring that tasks are executed in the correct order and without circular dependencies.

[0018] Multimodal Large Language Model (MLLM): This refers to an AI model capable of processing multiple modalities of data, including text, images, audio, and video. In this application embodiment, it is used for character extraction, storyboard generation, image generation, audio generation, and video clip generation, respectively. The multimodal large model may include: an LLM (Large Language Model) for text processing, and large image models, large audio models, and large video models implemented based on and extended from the LLM.

[0019] Segmented debugging refers to a mechanism that allows pausing and inputting segmented debugging commands at any task during the execution of the video generation workflow. The debugging mechanism then re-executes the target task and downstream tasks indicated by the segmented debugging command only for the text segment corresponding to the target content indicated by the segmented debugging command in the target text. This eliminates the need to re-execute the entire video generation workflow for the entire target text, which can significantly improve video optimization efficiency.

[0020] Production Version: This refers to the identifier of the output data obtained from multiple executions of the same task during multiple video generation processes for the same target text. It supports quality comparison and rollback operations between different versions of output data, ensuring the traceability and controllability of the video generation process.

[0021] Audio-Video Synchronization Rate: This refers to the degree of matching between video and audio on the timeline. In this application embodiment, it is required to reach more than 99% to ensure the smoothness of the target video.

[0022] TTS (Text-to-Speech): refers to the technology of converting text into audio, which is used in this application embodiment to generate dialogue audio and narration audio for comic book characters.

[0023] FFmpeg: refers to an open-source multimedia processing framework that supports functions such as video splicing, transition effects, subtitle overlay, audio synthesis, and format conversion. In this application embodiment, FFmpeg is used to synthesize video clips and audio from multiple storyboard texts into a target video.

[0024] This application embodiment executes the above-described video generation workflow to generate a target video, which may be an AI-generated comic based on a novel text. Figure 1 This is a schematic diagram of a hardware and software layered framework provided in an embodiment of this application. (Refer to...) Figure 1As shown, the first terminal 101 is the terminal device that triggers the video generation request, and the first server 102 is the background server corresponding to the first terminal 101. The first server 102 is wirelessly connected to the first terminal 101. In order to realize the configuration and execution of the video generation workflow, the first terminal 101 is equipped with a workflow platform layer (pioneer-admin) 107, and the first server 102 is equipped with a workflow orchestration service layer (pioneer-workflow-api) 106, a workflow execution service layer (pioneer-workflow-executor) 105, an engine layer (pioneer-engine) 104, and a task interface layer 103.

[0025] Each layer in the aforementioned first server 102 can be set up in the same first server 102 (referred to as the central node) to achieve a centralized service architecture. This centralized service architecture has good data synchronization, good data consistency, and high data security, but the central node has a high load pressure. Of course, the layers in the aforementioned first server 102 can also be distributed and set up in multiple first servers 102 (referred to as edge nodes) to achieve distributed deployment, with a central node set up. Each edge node runs independently, and the central node is used for task scheduling, which can reduce the load pressure on the central node and improve operating efficiency. However, distributed deployment has problems with poor data synchronization, data consistency, and data security. In practical applications, the deployment method can be selected according to actual needs, and this application embodiment does not impose any restrictions on it.

[0026] The aforementioned workflow platform layer 107 provides an interactive interface for workflow configuration. Users can configure the video generation workflow and obtain workflow configuration information through the interface. The workflow configuration information may include the number of tasks in the video generation workflow, task attributes, and dependencies between tasks.

[0027] Workflow orchestration service layer 106 is used to convert workflow configuration information into video generation workflows represented by DAG.

[0028] The workflow execution service layer 105 is used to call the execution node 1041 in the engine layer 104 to execute tasks in the video generation workflow according to the video generation request.

[0029] Engine layer 104 includes multiple execution nodes 1041, each used to execute different tasks in the video generation workflow. Specifically, each execution node 1041 can call the task interface 1031 corresponding to that execution node 1041 in task interface layer 103 to execute a task. Engine layer 104 can also re-execute some tasks and record the output data of each task.

[0030] The task interface layer 103 includes two types of task interfaces 1031: large model (pioneer-llm) 10310 and FFmpeg tools (pioneer-tools) 10311. The large model 10310 specifically includes multimodal large models such as: LLM model 10312, image large model 10313, audio large model 10314, and video large model 10315. Here, the large model can be a virtual model interface that encapsulates multiple models to shield the differences between them.

[0031] based on Figure 1 The hardware and software layered framework shown Figure 2 This is a schematic diagram illustrating the configuration and execution process of a video generation workflow provided in an embodiment of this application.

[0032] During the configuration of the video generation workflow, the first terminal 101 executes step 1011, calling the workflow platform layer 107 to generate a workflow configuration request, and step 1012, sending the workflow configuration request to the first server 102. The first server 102 executes step 1021, calling the workflow orchestration service layer 106 to convert the workflow configuration information in the workflow configuration request into a DAG-represented video generation workflow, and step 1022, sending configuration result information to the first terminal 101. The configuration result information indicates whether the configuration was successful.

[0033] During the execution of the video generation workflow, the first terminal 101 executes step 1013, sending a video generation request to the first server 102. The first server 102 executes step 1023, sequentially calling the workflow execution service layer 105, engine layer 104, and task interface layer 103 to execute each task in the video generation workflow, obtaining the target video, and then in step 1024, sending the target video to the first terminal 101.

[0034] In some application scenarios, the first terminal 101 can also send the target video to other terminals for playback. (See reference...) Figure 2 As shown, the second terminal 109 is a terminal device for playing the target video, and the second server 108 is a server that provides the target video. The second server 108 is wirelessly connected to the first terminal 101 and the second terminal 109. The process of the first terminal 101 sending the target video to the other terminals for playback includes: the first terminal 101 executing step 1014, sending the target video to the second server 108; the second terminal 109 executing step 1091, requesting the target video; and the second server 108 executing step 1081, sending the target video to the second terminal 109, so that the second terminal 109 can play the target video.

[0035] The first server 102 and the second server 108 mentioned above can be two different servers or the same server. For example, both the first server 102 and the second server 108 are video servers corresponding to the same video application. The first terminal 101 used by user 1 and the second terminal 109 used by user 2 both have video applications installed. User 1 can trigger a video generation request in the video application on the first terminal 101, causing the video server to generate the target video. User 2 can watch the target video in the video application on the second terminal 109.

[0036] Understandably, in Figure 1 and Figure 2 The video generation method in this application embodiment is executed by the first server 102 described above. However, in practical applications, the layer in the first server 102 can also be deployed in the first terminal 101 or local device so that the first terminal 101 or local device can execute the video generation method in this application embodiment.

[0037] The first terminal 101 and the second terminal 109 can be the same terminal device or different terminal devices. Terminal devices can include, but are not limited to, cloud devices, servers, mobile phones, computers, tablets, etc.

[0038] It should be noted that the requests, instructions, execution information, user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0039] Reference Figure 3 The diagram illustrates a flowchart of a video generation method provided in an embodiment of this application, including: Step 201: Obtain the target text and video generation workflow. The video generation workflow includes multiple tasks with dependencies, and the target text includes at least one text segment.

[0040] Step 202: Execute the video generation workflow for the target text and record the output data of the task for at least one text segment.

[0041] Step 203: Upon receiving the segmented debugging instruction, determine the re-execution text segment and the re-execution task; the re-execution text segment includes the text segment in the target text corresponding to the target content indicated by the segmented debugging instruction; the re-execution task includes the target task indicated by the segmented debugging instruction and the downstream task of the target task in the video generation workflow.

[0042] Step 204: Based on the output data of the direct dependent tasks of the re-execution task and the re-execution text segment, re-execute the re-execution task, and obtain the target video corresponding to the target text when the video generation workflow is completed.

[0043] In some possible implementations, steps 201 to 204 described above can be performed by... Figure 1 The workflow shown is executed at the service layer.

[0044] The target text can be a novel, with the AI-generated animated series based on that novel serving as the target video. The target text is input by the user in the client on the first terminal. Of course, the target text can also be any other type of text.

[0045] The video generation workflow is used to generate target videos from target text. Different target texts can reuse the same video generation workflow, or different video generation workflows can be used. The tasks included in the video generation workflow are executed sequentially until the last task, which outputs the target video. Figure 4 This is a schematic diagram of the DAG structure of a video generation workflow provided in an embodiment of this application. (Refer to...) Figure 4 As shown, the video generation workflow includes the following tasks: character extraction task 21, storyboard task 22, cue word generation task 23, character diagram generation task 24, storyboard diagram generation task 26, audio generation task 27, sound library generation task 25, video clip generation task 28, and video compositing task 29. Each task in the above video generation workflow will be explained below.

[0046] The character extraction task 21 is used to extract character feature text from the target text. Character feature text includes, but is not limited to, character name, physical characteristics, and personality attributes. In some implementations, the character extraction task 21 can invoke an LLM to extract character feature text.

[0047] Storyboard task 22 is used to divide the target text into multiple storyboard texts. In some implementations, storyboard task 22 may invoke an LLM to divide the target text into multiple text segments according to a narrative structure of "introduction, development, climax, and conclusion," and further divide each text segment into multiple storyboard texts. The text segments here may include at least one paragraph and / or at least one chapter. For example, the target text may be divided into 6 to 8 chapters, each chapter as a text segment, and each chapter further divided into 25 to 30 storyboard texts.

[0048] The prompt generation task 23 is used to generate character prompts based on character feature text, generate storyboard prompts based on storyboard text, and extract audio text (TTS) from storyboard text. In some implementations, the prompt generation task 23 can call an LLM to generate character prompts, storyboard prompts, and audio text. Character prompts are prompts used to control the generation of character images from the image model. Character feature text is structured data, and character prompts are natural language text generated based on this structured data, facilitating the character image generation task 24 to call the image model to understand the input information and accurately generate character images. For example, character prompts could be: Zhang San, male, 30 years old, short black hair, sword-like eyebrows, wearing ancient swordsman attire. Storyboard prompts include scene-related keywords from the storyboard text. Audio text includes dialogue and narration from the storyboard text. For example, the storyboard text is: "The candlelight flickers indoors, Zhang San angrily questions Li Si: Why did you betray me?", the storyboard cues are: "Indoor scene, night, candlelight, Zhang San angrily questions, Li Si stands opposite him," and the audio text is: "Why did you betray me?". Character image generation task 24 is used to generate character images based on the character cues. Specifically, character image generation task 24 can call large image models to generate character images. Character images can include images of the character from multiple angles, such as front view, side view, and back view. The generated character images can be stored using the character name as a unique identifier for easy retrieval when generating storyboard images.

[0049] Storyboard generation task 26 is used to generate at least one storyboard based on storyboard cues, the corresponding scene style reference image, and character images. Specifically, storyboard generation task 26 can use large image models to generate storyboards. The scene style reference image is related to the scene indicated by the storyboard cues and can be obtained by matching from a scene style reference image library. The scene style reference image library stores at least one scene style reference image and the corresponding tag text for each scene style reference image. Therefore, the storyboard cues can be matched with the tag text, and the scene style reference image that matches the tag text with the storyboard cues can be used as the scene style reference image for that storyboard cues. For example, the scene style reference image library includes: scene style reference image P1 corresponding to the tag text "ancient style interior, candlelight lighting, warm color tone, wooden furniture", scene style reference image P2 corresponding to the tag text "ancient style exterior, moonlit night, cool color tone, courtyard, tree shadows", and scene style reference image P3 corresponding to the tag text "ancient style exterior, rainy day, dark color tone, stone slab road, oil paper umbrella". For the storyboard cues "Indoor scene, night, candlelight, Zhang San angrily questioning, Li Si standing opposite," the scene style reference image obtained by matching "indoor scene, night, candlelight" in the storyboard cues is P1 as described above. The large image model can first extract the corresponding scene style reference image from the scene style reference image library based on the storyboard cues, and then combine the storyboard cues, the corresponding scene style reference image, and the character image to generate a storyboard image, ensuring that the storyboard image includes scene content, character actions, and expressions. The character image generated using character feature text is used as a background image reference for the storyboard image, ensuring consistency of the character across different storyboard images even when the actions and emotions of the same character differ. Furthermore, the scene style reference image is used as a background image reference to ensure the accuracy and consistency of the scene style in the storyboard image.

[0050] The voice library generation task 25 generates voice features for a character based on the character feature text, and adds the voice features and the character's name from the character feature text to the character voice library. The voice library generation task 25 can call an LLM (Local Level Management) to generate voice features. A character's voice features may include timbre, speech rate, volume, and exemplary audio clips. This embodiment of the application can generate voice features for different characters based on their character features, making the character's dialogue audio more suitable for the character.

[0051] Audio generation task 27 is used to generate audio from audio text. Specifically, audio generation task 27 can call a large audio model to generate audio.

[0052] Video clip generation task 28 is used to generate video clips of the storyboard text based on at least one storyboard diagram. Specifically, video clip generation task 28 calls a large video model to generate dynamic video clips of 5 to 10 seconds.

[0053] Video compositing task 29 is used to combine video clips and audio from multiple storyboard texts into a target video. Specifically, FFmpeg can be used to perform video compositing. This first synthesizes video clips according to the storyboard text, and then combines the video clips from multiple storyboard texts into the target video, realizing segmented and progressive video compositing, which helps to ensure audio-visual synchronization.

[0054] From the above Figure 4 As can be seen, task 23 for generating cue words directly depends on task 21 for extracting characters and task 22 for storyboarding; task 24 for generating character diagrams directly depends on task 23 for generating cue words; task 25 for generating a sound library directly depends on task 21 for extracting characters; task 26 for generating storyboard diagrams directly depends on task 24 for generating character diagrams and task 23 for generating cue words; task 27 for generating audio directly depends on task 23 for generating cue words and task 25 for generating a sound library; task 28 for generating video clips directly depends on task 26 for generating storyboard diagrams; and task 29 for compositing video directly depends on task 28 for generating video clips and task 27 for generating audio.

[0055] Based on the above dependencies, the upstream tasks of any task include the remaining tasks that the task directly or indirectly depends on. Therefore, character extraction task 21 and storyboard task 22 have no upstream tasks. The upstream tasks of cue word generation task 23 include character extraction task 21 and storyboard task 22. The upstream task of sound library generation task 25 includes character extraction task 21. The upstream tasks of character graph generation task 24 include: character extraction task 21, storyboard task 22, and cue word generation task 23. The upstream tasks of storyboard graph generation task 26 include: character graph generation task 24, character extraction task 21, storyboard task 22, and cue word generation task 23. The upstream tasks of audio generation task 27 include: character extraction task 21, storyboard task 22, cue word generation task 23, and sound library generation task 25. The upstream tasks of video clip generation task 28 include: storyboard graph generation task 26, character graph generation task 24, character extraction task 21, storyboard task 22, and cue word generation task 23. The upstream tasks of video compositing task 29 include... Figure 4 All tasks except for medium-length video compositing task 29.

[0056] Based on the above dependencies, downstream tasks of any task include other tasks that directly or indirectly depend on that task. Therefore, video compositing task 29 has no downstream tasks. Downstream tasks of character extraction task 21 include: cue word generation task 23, sound library generation task 25, character graph generation task 24, storyboard generation task 26, audio generation task 27, video clip generation task 28, and video compositing task 29. Downstream tasks of storyboard generation task 22 include: cue word generation task 23, character graph generation task 24, storyboard generation task 26, audio generation task 27, video clip generation task 28, and video compositing task 29. Downstream tasks of cue word generation task 23 include: character graph generation task 24, storyboard generation task 26, audio generation task 27, video clip generation task 28, and video compositing task 29. Downstream tasks of sound library generation task 25 include: audio generation task 27 and video compositing task 29. Downstream tasks of character graph generation task 24 include: storyboard generation task 26, video clip generation task 28, and video compositing task 29. Downstream tasks of storyboard generation task 26 include video clip generation task 28 and video compositing task 29. Downstream tasks of audio generation task 27 include video compositing task 29. Downstream tasks of video clip generation task 28 include video compositing task 29.

[0057] In some possible implementations, the above Figure 4 The cue word generation task 23 shown can be further divided into three tasks: character cue word generation task, storyboard cue word generation task, and audio text generation task. The character cue word generation task is used to generate character cue words based on character feature text, the storyboard cue word generation task is used to generate storyboard cue words based on storyboard text, and the audio text generation task is used to generate audio text based on storyboard text.

[0058] Based on this, the character cue word generation task directly depends on the character extraction task 21, the storyboard cue word generation task and the audio text generation task directly depend on the storyboard task 22, the character diagram generation task 24 directly depends on the character cue word generation task, the sound library generation task 25 directly depends on the character extraction task 21, the storyboard diagram generation task 26 directly depends on the character diagram generation task 24 and the storyboard cue word generation task, the audio generation task 27 directly depends on the audio text generation task and the sound library generation task 25, the video clip generation task 28 directly depends on the storyboard diagram generation task 26, and the video compositing task 29 directly depends on the video clip generation task 28 and the audio generation task 27.

[0059] Based on the above dependencies, the upstream tasks of the character cue word generation task include character extraction task 21. The upstream tasks of both the storyboard cue word generation task and the audio text generation task are storyboard task 22. The upstream task of the sound library generation task 25 includes character extraction task 21. The upstream tasks of the character graph generation task 24 include: character extraction task 21 and character cue word generation task. The upstream tasks of the storyboard graph generation task 26 include: character graph generation task 24, character cue word generation task, character extraction task 21, storyboard task 22, and storyboard cue word generation task. The upstream tasks of the audio generation task 27 include: character extraction task 21, storyboard task 22, audio text generation task, and sound library generation task 25. The upstream tasks of the video clip generation task 28 include: storyboard graph generation task 26, character graph generation task 24, character extraction task 21, storyboard task 22, character cue word generation task, and storyboard cue word generation task. The upstream tasks of the video compositing task 29 include... Figure 4 All tasks except for medium-length video compositing task 29.

[0060] Based on the aforementioned dependencies, the downstream tasks of character extraction task 21 include: character cue word generation task, sound library generation task 25, character diagram generation task 24, storyboard diagram generation task 26, audio generation task 27, video clip generation task 28, and video compositing task 29. The downstream tasks of storyboard task 22 include: storyboard cue word generation task, audio text generation task, storyboard diagram generation task 26, audio generation task 27, video clip generation task 28, and video compositing task 29. The downstream tasks of character cue word generation task include: character diagram generation task 24, storyboard diagram generation task 26, video clip generation task 28, and video compositing task 29. The downstream tasks of storyboard cue word generation task include: storyboard diagram generation task 26, video clip generation task 28, and video compositing task 29. The downstream tasks of audio text generation task include: audio generation task 27 and video compositing task 29. The downstream tasks of sound library generation task 25 include: audio generation task 27 and video compositing task 29. Downstream tasks of character image generation task 24 include: storyboard image generation task 26, video clip generation task 28, and video compositing task 29. Downstream tasks of storyboard image generation task 26 include: video clip generation task 28 and video compositing task 29. Downstream tasks of audio generation task 27 include: video compositing task 29. Downstream tasks of video clip generation task 28 include: video compositing task 29.

[0061] As can be seen from the above description of the video generation workflow, the embodiments of this application divide the video generation process into multiple tasks. Compared with using a single machine model to directly generate the target video from the target text end-to-end, the embodiments of this application facilitate segmented debugging, allow intervention in any intermediate task, ensure the output data quality of each task, and help improve the generation quality of long videos.

[0062] Reference Figure 4 As shown, the tasks in the video generation workflow described above can be divided into the following task groups according to the type of their output data: text generation task group 213, image generation task group 214, sound generation task group 215, and video generation task group 216. Specifically, text generation task group 213 includes: character extraction task 21, storyboard task 22, and cue word generation task 23; text generation task group 213 is used to generate text. Image generation task group 214 includes: character image generation task 24 and storyboard image generation task 26; image generation task group 214 is used to generate images. Sound generation task group 215 includes: audio generation task 27 and sound library generation task 25; sound generation task group 215 is used to generate audio. Video generation task group 216 includes: video clip generation task 28 and video compositing task 29; video generation task group 216 is used to generate video.

[0063] In some embodiments, each task in the video generation workflow described above can be deployed as a microservice. Compared to the conventional approach of implementing each task through a serverless function, deploying tasks as microservices has the following advantages: First, microservices overcome the limitations of function execution time; second, microservice instances run for a long time by default and are not destroyed even without requests, with only a small number of instances being initialized during deployment scaling, thus reducing the probability of cold starts, decreasing cold start latency, and improving task execution timeliness; third, the default long-term operation of microservice instances makes the state management complexity of microservices lower than that of cross-function state management, facilitating segmented re-execution and version rollback.

[0064] After obtaining the above video generation workflow, step 202 is executed, which specifically includes: Step 2021: Execute independent tasks in the video generation workflow in parallel.

[0065] Step 2022: Execute the tasks with dependencies in the video generation workflow in sequence.

[0066] In some possible implementations, the above steps 2021 to 2022 can be performed by the workflow execution service layer.

[0067] Independent tasks are those without direct or indirect dependencies, while dependent tasks include those with direct or indirect dependencies. When executing step 2022, executing any task followed by its downstream tasks can also be achieved through the following process: When executing each task, determine if it has an upstream task. If no upstream task exists, execute the task; if an upstream task exists, execute the upstream task.

[0068] In conventional implementation, Figure 4 In the video generation workflow shown, each task is executed by a corresponding task interface called by an execution node in the engine layer. Different execution nodes execute different tasks, and the task interfaces for different tasks may be the same or different. For example, Figure 4 The execution node for the character extraction task 21 is called the character extraction node; the execution node for the storyboard task 22 is called the storyboard node; the execution node for the cue word generation task 23 is called the cue word generation node; the execution node for the sound library generation task 25 is called the sound library generation node; the execution node for the character graph generation task 24 is called the character graph generation node; the execution node for the storyboard graph generation task 26 is called the storyboard graph generation node; the execution node for the audio generation task 27 is called the audio generation node; the execution node for the video clip generation task 28 is called the video clip generation node; and the execution node for the video compositing task 29 is called the video compositing node. The interfaces for these tasks can be either large models or the FFmpeg tool. Specifically, the task interfaces for character extraction tasks 21, storyboard tasks 22, and cue word generation tasks 23 are LLM; the task interfaces for character graph generation tasks 24 and storyboard graph generation tasks 26 are large image models; the task interface for audio generation task 27 is a large audio model; the task interface for sound library generation task 25 can be either LLM or a large audio model; the task interface for video clip generation task 28 is a large video model; and the task interface for video compositing task 29 is the FFmpeg tool.

[0069] The above tasks can be performed in the following order: Step 301: Execute the role extraction task.

[0070] Step 302: Perform the storyboard task.

[0071] Step 303: Perform the prompt word generation task.

[0072] Step 304: Execute the sound library generation task.

[0073] Step 305: Execute the character image generation task.

[0074] Step 306: Execute the storyboard generation task.

[0075] Step 307: Execute the audio generation task.

[0076] Step 308: Execute the video clip generation task.

[0077] Step 309: Perform the video compositing task.

[0078] However, in this embodiment of the application, in order to improve the execution speed of the video generation workflow, refer to Figure 5 As shown, steps 301 to 309 can be executed in the following order: first, steps 301 and 302 are executed in parallel, then steps 303 and 304 are executed in parallel; after executing step 303, a group of steps and step 307 are executed in parallel, wherein executing a group of steps includes: executing steps 305, 306 and 308 in sequence; finally, step 309 is executed.

[0079] As can be seen, the embodiments of this application can execute independent tasks in parallel, which helps to improve the execution speed of the video generation workflow. When the production chain involves hundreds of large model calls, the complexity of data flow and state management between tasks increases exponentially. The traditional serial execution mode cannot support large-scale concurrent tasks. The parallel execution of some independent tasks in this application helps to improve the execution speed.

[0080] After each task is completed in step 202, the following steps are performed: Step 2023: Record the text identifier of the target text, the task identifier of the task, the text segment identifier of the text segment, and the output data of the task to the text segment according to the corresponding relationship.

[0081] In some possible implementations, step 2023 described above may be performed by the workflow execution service layer.

[0082] Compared to storing data as in-memory variables, the records here are persistent, or non-volatile, to ensure that the output data persists even after memory is cleared, thus providing the conditions for re-execution in step 2041 below. Persistent storage can be done in databases such as Redis / MySQL. The target text's text identifier is used to track the entire video generation task flow, while the task identifier is used to track each task within the target text's video generation task. Therefore, when a segmented debugging instruction is triggered for any target text, the corresponding output data can be obtained based on the target text's text identifier indicated by the segmented debugging instruction.

[0083] Based on the above step 2023, step 203 includes: Step 2031: Extract the re-execution text identifier from the segmented debugging instructions; determine the re-execution text segment identifier and the re-execution task identifier.

[0084] In some possible implementations, step 2031 described above can be performed by the workflow execution service layer.

[0085] Based on step 2023 above, step 204 includes: Step 2041: For any re-execution task, based on the re-execution text identifier, the re-execution task identifier, and the re-execution text segment identifier, obtain the output data of the re-execution text segment from the output data of the directly dependent tasks of the re-execution task, and use it as the input data of the re-execution task to re-execute the re-execution task.

[0086] In some possible implementations, step 2041 described above may be performed by the workflow execution service layer.

[0087] Among them, the re-execution text identifier is the text identifier of the target text that needs to be partially re-executed, the re-execution task identifier is the task identifier of the re-execution task, and the re-execution text segment identifier is the text segment identifier of the re-execution text segment. The text segment identifier of the text segment corresponding to the target content indicated by the segmented debugging instruction in the target text is used as the re-execution text segment identifier, and the task identifier of the target task indicated by the segmented debugging instruction and the task identifier of the downstream task of the target task in the video generation workflow are used as the re-execution task identifier.

[0088] The direct dependencies of the aforementioned re-execution task include at least one task that the re-execution task directly depends on. For example, refer to... Figure 4 As shown, the direct dependent tasks of storyboard generation task 26 include: character image generation task 24 and cue word generation task 23.

[0089] Since the output data of each task includes the output data of all text segments for all target texts, it is necessary to obtain the output data of the re-executed text segments of the target text from the output data of the tasks that directly depend on the re-execution task, so that the re-execution task is performed only for the re-executed text segments. Thus, this embodiment of the application achieves the re-execution of partial tasks and partial text segments. Compared to the re-execution of the entire video generation workflow and the entire target text, this embodiment of the application can effectively improve the re-execution speed and optimization efficiency.

[0090] In some possible implementations, based on step 2023, the above method further includes the following steps: Step 2024: Record the version information of the output data. The version information is used to distinguish the multiple output data of the text segment by the task.

[0091] Based on steps 2024 and 2041, the specific steps include: Step 20411: Obtain the first version information selected from at least one version information.

[0092] Step 20412: Extract the first version information and the output data corresponding to the re-execution text segment from the output data of the direct dependent tasks of the re-execution task as the input data of the re-execution task.

[0093] In some possible implementations, steps 2024, 20411, and 20412 may be performed by the workflow execution service layer.

[0094] The version information mentioned above can be represented by the output time of the output data or by a number. For example, if the storyboard generation task is executed twice at times t1 and t2, generating two storyboards D1 and D2 for text segment B1 respectively, then the version information of the storyboards D1 and D2 output by the storyboard generation task for text segment B1 is time t1 and time t2, or numbers 01 and 02 respectively.

[0095] For any re-execution task, the first version information is one of the version information of multiple output data obtained from multiple executions of the directly dependent task of the re-execution task. The output data corresponding to the first version information is used as the input data of the re-execution task, which improves the re-execution flexibility of the re-execution task.

[0096] In some possible implementations, step 2024 further includes: Step 205: Obtain second version information.

[0097] Step 206: Select the output data corresponding to the second version information from the output data of the last task as the target video.

[0098] In some possible implementations, steps 205 and 206 may be performed by the workflow execution service layer.

[0099] The second version information refers to the version selected for the target video, and the last task is... Figure 4 The video synthesis task 29 shown has no downstream tasks, and its output data is the target video. This embodiment of the application can implement version rollback of the target video, allowing any version of the target video to be selected as the final generated target video.

[0100] In some possible implementations, step 202 specifically includes: Step 2025: Generate execution information for multiple tasks related to the target text based on the target text and video generation workflow, and add the execution information of multiple tasks to the execution queue of the execution node corresponding to the task in the engine layer so that the execution node can execute the task. The execution information includes: task attributes and task input data. The input data of the first task is the target text, and the input data of the remaining tasks are the output data of the tasks that directly depend on the remaining tasks.

[0101] In some possible implementations, step 2025 may be performed by the workflow execution service layer.

[0102] The target text is extracted by the workflow execution service layer from the received video generation request. In some possible implementations, the video generation request may also include a workflow identifier of the video generation workflow used by the target text, so as to determine the corresponding video generation workflow based on the workflow identifier, or the video generation request may also include the video generation workflow used by the target text.

[0103] The workflow execution service layer can extract task attributes from the tasks in the video generation workflow. First, the target text and the attributes of the first task are added to the execution queue of the corresponding execution node in the engine layer. Then, each time a task finishes execution, its output data is added to the execution queue of the remaining tasks that depend on it, until the last task in the video generation workflow finishes execution and the target video of the target text is obtained. The execution node extracts the task attributes and input data from the execution queue to extract the task interface and execution parameters from the task attributes, and then calls the task interface to execute the task based on the execution parameters and input data.

[0104] It is understandable that each execution node in the engine layer corresponds to an execution queue, which stores the execution information of tasks of the same type to be executed by that execution node, so that the execution node executes tasks one by one in a first-in-first-out order. Tasks whose execution information is added to the queue first are executed first, and tasks whose execution information is added later are executed later. For example, refer to... Figure 6As shown, for any execution node, the execution information RQ(TT1), RQ(TT2), RQ(TT3), and RQ(TT4) of the tasks for target texts TT1, TT2, TT3, and TT4 are sequentially added to the execution queue of that execution node. The execution information RQ(TT1) of the task for target text TT1 is at the head of the queue, and the execution information RQ(TT4) of the task for target text TT4 is at the tail of the queue. Therefore, the execution information RQ(TT1) of the task for target text TT1 is extracted from the head of the queue first, and the task is executed for target text TT1. The execution information RQ(TT2) of the task for target text TT2 is extracted from the head of the queue second, and the task is executed for target text TT2. The execution information RQ(TT3) of the task for target text TT3 is extracted from the head of the queue third, and the task is executed for target text TT3. The execution information RQ(TT4) of the task for target text TT4 is extracted from the head of the queue fourth, and the task is executed for target text TT4.

[0105] As can be seen, the embodiments of this application ensure the accurate scheduling order of multiple target texts through the pending execution queue. Furthermore, the target text and workflow identifier in the target video generation request are input by the user when triggering the target video generation request on the first terminal, allowing the user to flexibly select the video production workflow and target text.

[0106] In some possible implementations, some or all of the above tasks can be configured with a quality inspection interface; therefore, the above method further includes: Step 207: After executing each task, call the task's quality inspection interface to check the quality of the task's output data and obtain the quality inspection result.

[0107] Step 208: If the quality inspection results of the task do not meet the preset conditions of the task, adjust the execution parameters of the task to re-execute the task.

[0108] In some possible implementations, steps 207 and 208 may be performed by the workflow execution service layer.

[0109] The implementation process of step 208 includes: if the quality detection result of a task does not meet the preset conditions of the task, and if the number of re-executions of the task is less than the preset number, then the execution parameters of the task are adjusted to re-execute the task; if the number of re-executions of the task is greater than or equal to the preset number, then the task is not re-executed, and the final output data of the task is selected from the output data obtained from multiple executions of the task based on the quality detection result. Steps 207 and 208 are implemented during the execution of step 202, or they can be implemented during the execution of step 204. Specifically, after each task is executed, if the task has a quality detection interface, then the quality detection interface is called to detect the quality of the output data and obtain the quality detection result.

[0110] It is understandable that if the quality detection result of a task meets the preset conditions of that task and there are downstream tasks, then the downstream tasks continue to be executed; if the quality detection result of a task meets the preset conditions of that task and there are no downstream tasks, then the execution of the video generation workflow ends. This application embodiment can ensure the quality of the task's output data through the aforementioned quality detection interface, for example, ensuring the quality of character feature text, storyboard text, character prompts, storyboard prompts, audio text, character images, storyboard images, and video clips, thereby ensuring the quality of the target video. Both the aforementioned quality detection interface and preset conditions can be flexibly configured.

[0111] For example, a quality inspection interface can be configured for the following tasks in the video generation workflow: character graph generation task, storyboard generation task, audio generation task, and video clip generation task. Accordingly, Figure 7 It shows in Figure 5 The following additional steps are required based on the existing steps: For a character graph generation task configured with a quality inspection interface, after executing step 305, proceed to step 310 to determine if the character graph quality is acceptable. If the quality inspection result obtained by the character graph generation task's quality inspection interface meets the preset conditions of the character graph generation task, then the character graph quality is deemed acceptable, and step 306 can be executed. If the character graph quality inspection result does not meet the preset conditions of the character graph generation task, then the character graph quality is deemed unacceptable, and step 305 needs to be re-executed until the character graph quality is acceptable or the number of times step 305 is executed is greater than or equal to the preset number of times.

[0112] For storyboard generation tasks configured with a quality inspection interface, step 311 is executed after step 306 to determine if the storyboard quality is acceptable. If the quality inspection result obtained by the quality inspection interface of the storyboard generation task meets the preset conditions of the storyboard generation task, the storyboard quality is deemed acceptable, and step 308 can be executed. If the quality inspection result of the storyboard does not meet the preset conditions of the storyboard generation task, the storyboard quality is deemed unacceptable, and step 306 needs to be re-executed until the storyboard quality is acceptable or the number of times step 306 is executed is greater than or equal to the preset number of times.

[0113] For audio generation tasks configured with an audio detection interface, after executing step 307, proceed to step 312 to determine if the audio quality is acceptable. If the quality detection result obtained by the audio generation task's quality detection interface meets the preset conditions of the audio generation task, then the audio quality is deemed acceptable, and step 309 can be executed. If the audio quality detection result does not meet the preset conditions of the audio generation task, then the audio quality is deemed unacceptable, and step 307 needs to be re-executed until the audio quality is acceptable or the number of times step 307 is executed is greater than or equal to the preset number.

[0114] For video clip generation tasks configured with a quality inspection interface, step 313 is executed after step 308 to determine whether the video clip quality is acceptable. If the quality inspection result obtained by the quality inspection interface of the video clip generation task meets the preset conditions of the video clip generation task, then the video clip quality is deemed acceptable, and step 309 can be executed. If the quality inspection result of the video clip does not meet the preset conditions of the video clip generation task, then the video clip quality is deemed unacceptable, and step 308 needs to be re-executed until the video clip quality is acceptable or the number of times step 308 is executed is greater than or equal to the preset number of times.

[0115] In some possible implementations, the preset conditions for the storyboard generation task include at least one of the following: the clarity of the storyboard is greater than or equal to a preset clarity, and the matching degree between the storyboard and the storyboard prompt is greater than or equal to a preset matching degree.

[0116] The preset conditions for the character image generation task include at least one of the following: the matching degree between the character image and the character prompt is greater than or equal to the preset matching degree, and the clarity of the character image is greater than or equal to the preset clarity.

[0117] The preset conditions for the audio generation task include at least one of the following: the percentage difference between the audio duration and the corresponding audio text reading duration is less than or equal to a first preset percentage; and the percentage of time the audio volume is within a preset volume range is greater than or equal to a second preset percentage. Here, the percentage difference is the ratio of the difference between the audio duration and the audio text reading duration to the audio duration itself, and the percentage of duration is the ratio of the duration of the audio volume within the preset volume range to the total audio duration.

[0118] The preset conditions for the video clip generation task include at least one of the following: the inter-frame pixel difference of the video clip is greater than a preset pixel difference, and the optical flow motion of the video clip is greater than a preset optical flow motion. These preset conditions are used to determine whether the motion of the video clip is sufficient. When the motion of the video clip is sufficient, the video clip quality is considered acceptable; when the motion of the video clip is insufficient or even static, the video clip quality is considered unacceptable.

[0119] For any image in a character illustration or storyboard, the sharpness can be represented by the grayscale difference of the image. Sharpness is positively correlated with the grayscale difference; the greater the grayscale difference, the higher the sharpness.

[0120] The matching degree between images and prompts can be achieved through the following steps: perform semantic analysis on the images using LLM to obtain image descriptions, and calculate the similarity between the image descriptions and prompts as the matching degree.

[0121] This application embodiment can assess the quality of images, including the quality of storyboards and character images, by evaluating the matching degree of the images and prompts, as well as the clarity of the images, thereby ensuring the quality of video clips and target videos.

[0122] The inter-frame pixel difference in the preset conditions of the above video clip generation task is the average pixel difference of all adjacent storyboard images of the video clip. Specifically, for any two adjacent storyboard images of a video clip, the pixel difference at the corresponding position in the adjacent storyboard images is counted, and the pixel differences at all positions are summed to obtain the total pixel difference between the two adjacent storyboard images; then, the total pixel difference of all adjacent storyboard images of the video clip is averaged to obtain the inter-frame pixel difference of the video clip.

[0123] The optical flow motion in the preset conditions of the above video clip generation task refers to the displacement of pixels in the storyboard image between consecutive adjacent frames, calculated by optical flow technology. This displacement includes the direction and distance of pixel movement and is used to represent the relative motion amplitude of objects or cameras.

[0124] In some possible implementations, video clips and audio of multiple storyboard texts are combined into a target video, specifically including: step 2026, combining video clips and audio of multiple storyboard texts into a composite video according to text segments, the composite video including at least one video clip; combining the composite video of at least one text segment into the target video, the at least one text segment including: re-executed text segment and non-re-executed text segment other than the re-executed text segment.

[0125] Optionally, step 2026 can be executed by the video compositing node in the engine layer by calling the FFmpeg tool.

[0126] In this embodiment, the synthesized video for each text segment is a video composed of video clips and audio clips from multiple storyboard texts within that text segment. The synthesized videos from multiple text segments are concatenated according to the order of the text segments within the target text. This application's segmented synthesis of the target video, compared to directly synthesizing video clips and audio clips from multiple storyboard texts, offers several advantages. First, it reduces resource consumption per synthesis, avoiding synthesis failure due to memory overflow. Second, segmented synthesis yields longer target videos; compared to videos generated in the prior art with durations within 2 to 3 minutes, the target videos of this application can reach over 30 minutes in length. Third, segmented synthesis also improves audio-visual synchronization.

[0127] When generating the composite video, you can also overlay dialogue subtitles, narration subtitles, transition effects, and background music corresponding to the audio text of the storyboard. The volume ratio of the background music and audio can be flexibly configured as an execution parameter of the video compositing task, or a default volume ratio can be used, for example, the audio volume is 70% and the background music volume is 30%. When the execution parameters of the video compositing task include a pre-configured resolution, the target video can be generated according to that resolution. When the execution parameters of the video compositing task do not include a pre-configured resolution, multiple resolutions such as 480P, 720P, and 1080P can be output as target videos for user selection.

[0128] In some possible implementations, video clips and audio clips of multiple storyboard texts are combined into a composite video according to the text segments, specifically including: Step 2027: Add silent audio before and / or after the audio of the storyboard text to obtain the storyboard audio; combine the video clips and storyboard audio of multiple storyboard texts according to the text segments to form a composite video. This avoids audio overlap and interference between different storyboard audios, thus helping to improve audio quality.

[0129] Optionally, step 2027 above can be executed by the video compositing node in the engine layer by calling the FFmpeg tool.

[0130] Among them, silent audio can be audio with no sound, or audio with a volume less than or equal to a preset volume threshold. The storyboard audio is the audio obtained by sequentially splicing silent audio and audio of storyboard text, or the audio obtained by sequentially splicing audio of storyboard text and silent audio, or the audio obtained by sequentially splicing silent audio, audio of storyboard text and silent audio.

[0131] In a first possible implementation, for each storyboard text, the video clip and audio clip of the storyboard text are aligned chronologically and combined into a single video, which serves as the composite video of the storyboard text. Then, for a text segment, the composite videos corresponding to the multiple storyboard texts included in the text segment are concatenated into a single video according to the order of the storyboard texts in the text segment, which serves as the composite video of the text segment. Finally, the composite videos of each text segment are concatenated into the target video according to the order of the text segments in the target text.

[0132] In a second possible implementation, for each text segment, video clips of multiple storyboard texts included in the text segment are concatenated in the order of the storyboard texts to form a pure video of the text segment, and audio clips of multiple storyboard texts included in the text segment are concatenated in the order of the storyboard texts to form a pure audio of the text segment; then, the pure video clips of each text segment are concatenated in the order of the text segments in the target text to form a pure video of the target text, and the pure audio clips of each text segment are concatenated in the order of the text segments in the target text to form a pure audio of the target text; finally, the pure video clips and pure audio clips of the target text are time-aligned and synthesized into a target video.

[0133] In a third possible implementation, for each text segment, video clips of multiple storyboard texts included in the text segment are concatenated in the order of the storyboard texts to form a pure video of the text segment, and audio clips of multiple storyboard texts included in the text segment are concatenated in the order of the storyboard texts to form a pure audio clip of the text segment; then, the pure audio clips and pure video clips of each text segment are aligned and synthesized in time to form a composite video of the text segment; finally, the composite videos of each text segment are concatenated in the order of the text segments in the target text to form the target video.

[0134] For example, the target text is divided into text segment 1 and text segment 2. Text segment 1 is located after text segment 2. Text segment 1 includes storyboard text 1, storyboard text 2 and storyboard text 3. Text segment 2 includes storyboard text 4 and storyboard text 5.

[0135] In the first possible implementation described above, the video clip and audio of storyboard text 1 are first combined to form composite video 1 of storyboard text 1. Similarly, the video clip and audio of storyboard text 2 are combined to form composite video of storyboard text 2, the video clip and audio of storyboard text 3 are combined to form composite video of storyboard text 3, the video clip and audio of storyboard text 4 are combined to form composite video of storyboard text 4, and the video clip and audio of storyboard text 5 are combined to form composite video of storyboard text 5. Next, the composite videos corresponding to storyboard text 1, storyboard text 2, and storyboard text 3 are sequentially concatenated to form composite video of text segment 1, and the composite videos corresponding to storyboard text 4 and storyboard text 5 are sequentially concatenated to form composite video of text segment 2. Finally, the composite video of text segment 1 and the composite video of text segment 2 are sequentially concatenated to form the target video.

[0136] In the second possible implementation described above, firstly, the video clips of storyboard text 1, storyboard text 2, and storyboard text 3 are sequentially concatenated to form the pure video of text segment 1. Similarly, the audio clips of storyboard text 1, storyboard text 2, and storyboard text 3 are sequentially concatenated to form the pure audio of text segment 1. Likewise, the video clips of storyboard text 4 and storyboard text 5 are sequentially concatenated to form the pure video of text segment 2, and the audio clips of storyboard text 4 and storyboard text 5 are sequentially concatenated to form the pure audio of text segment 2. Next, the pure videos corresponding to text segments 1 and 2 are sequentially concatenated to form the pure video of the target text, and the pure audio clips corresponding to text segments 1 and 2 are sequentially concatenated to form the pure audio of the target text. Finally, the pure video and pure audio of the target text are time-aligned and synthesized into the target video.

[0137] In the third possible implementation described above, firstly, the video clips of storyboard text 1, storyboard text 2, and storyboard text 3 are sequentially concatenated to form the pure video of text segment 1. Similarly, the audio clips of storyboard text 1, storyboard text 2, and storyboard text 3 are sequentially concatenated to form the pure audio of text segment 1. Likewise, the video clips of storyboard text 4 and storyboard text 5 are sequentially concatenated to form the pure video of text segment 2, and the audio clips of storyboard text 4 and storyboard text 5 are sequentially concatenated to form the pure audio of text segment 2. Next, the pure video and pure audio clips of text segment 1 are combined to form the composite video of text segment 1, and the pure video and pure audio clips of text segment 2 are combined to form the composite video of text segment 2. Finally, the composite video of text segment 1 and the composite video of text segment 2 are sequentially concatenated to form the target video.

[0138] In some possible implementations, video clips and audio from multiple storyboard texts are combined into a target video, including: Step 2028: If the duration difference between the audio and video segments of the storyboard text is less than or equal to a preset duration difference, combine the video segments and audio of multiple storyboard texts into the target video. If the duration difference between the audio and video segments of the storyboard text is greater than a preset duration difference, regenerate video segments and / or regenerate audio for multiple storyboard texts until the duration difference between the audio and video segments of the storyboard text is less than or equal to the preset duration difference. This ensures audio-visual synchronization.

[0139] In some possible implementations, the condition judgment in step 2028 can be performed by the video synthesis node of the engine layer, and when the condition is met, the video synthesis node calls the FFmpeg tool to synthesize the video clips and audio of multiple storyboard texts into the target video.

[0140] In step 203 above, the segmented debugging instruction can be a text instruction. Therefore, semantic analysis can be performed on the text instruction to determine the target task and target content indicated by the segmented debugging instruction. Then, the target task and its downstream tasks are treated as re-executed tasks, and the text segment containing the target content is treated as the re-executed text segment.

[0141] For example, the segmented debugging command is: Adjust the storyboard images of Chapter 3, then according to Figure 4 As shown, the corresponding target task is storyboard generation task 26, and its downstream tasks include video clip generation task 28 and video compositing task 29. Therefore, the re-execution tasks include the following tasks: storyboard generation task 26, video clip generation task 28, and video compositing task 29. The re-executed text segment is the text segment containing Chapter 3. The storyboard prompts corresponding to the text segment containing Chapter 3 are input into storyboard generation task 26 to obtain the storyboard for the text segment containing Chapter 3. The storyboard corresponding to the text segment containing Chapter 3 is input into video clip generation task 28 to obtain video clips corresponding to each storyboard text in the text segment containing Chapter 3. The video clips corresponding to each storyboard text in the text segment containing Chapter 3 and the audio corresponding to each storyboard text in the text segment containing Chapter 3 are input into video compositing task 29. First, a composite video of the text segment containing Chapter 3 is obtained, which is then combined with the composite videos of the other text segments to form the target video.

[0142] For example, if the segmented debugging command is: "Adjust the character image of character A", then, referring to... Figure 4As shown, the corresponding target task is character graph generation task 24, and its downstream tasks include storyboard generation task 26, video clip generation task 28, and video compositing task 29. Therefore, the re-execution tasks include the following tasks: character graph generation task 24, storyboard generation task 26, video clip generation task 28, and video compositing task 29. The re-execution text segment is the text segment in which character A appears, so that character graph generation, storyboard generation task 26, video clip generation task 28, and video compositing task 29 are performed on the text segment in which character A appears.

[0143] In addition, the aforementioned segmented debugging instructions can also indicate the adjusted execution parameters of the target task, so that the target task can be re-executed according to the adjusted execution parameters.

[0144] In some possible implementations, the video generation workflow of this application embodiment is pre-configured by the user. Specifically, the video generation method further includes the following steps before step 201: Step 209: Receive a workflow configuration request. The workflow configuration request is used to configure the attributes and dependencies of multiple tasks. The attributes of a task include at least one of the following: the task interface, the task quality inspection interface, and the execution parameters of the task interface. The task interface is used to execute the task according to the execution parameters, and the quality inspection interface is used to perform quality inspection on the output data of the task.

[0145] Step 210: Generate video generation workflow based on workflow configuration request.

[0146] In some possible implementations, steps 209 and 210 may be performed by the workflow orchestration service layer.

[0147] The aforementioned workflow configuration request is generated by the configuration operations input by the user at the workflow platform layer. The workflow platform layer provides an interactive interface where users can create tasks and set their attributes and dependencies. The workflow orchestration service layer converts the workflow configuration information in the workflow configuration request into a video generation workflow represented by a DAG.

[0148] The execution parameters mentioned above are the relevant parameters of the task interface output data. Execution parameters for the character extraction task include: character feature type. Execution parameters for the storyboard task may include the number of storyboards and the length range of the storyboards. Execution parameters for the cue word generation task may include: the number and type of cue words. Execution parameters for the character image generation task include: the clarity, size, and number of character images. Execution parameters for the storyboard image generation task may include: the number of storyboard images corresponding to each storyboard text, the clarity of the storyboard images, and the size of the storyboard images. Execution parameters for the audio generation task may include: the volume, speech rate, timbre, and emotional tone of the narration audio. Execution parameters for the video clip generation task include: the frame rate of the video clip, the motion amplitude of adjacent frames, and the duration of the video clip; the motion amplitude can be: static, slight, moderate, or intense. Execution parameters for the video compositing task include: video format and duration.

[0149] In some possible implementations, the audio includes dialogue audio of at least one character, generated from audio text, specifically including: Step 2029: Extract at least one character's name and dialogue text from the audio text; extract the character's voice features from the character voice library based on the character's name in the audio text; generate the character's dialogue audio based on the voice features and the character's dialogue text.

[0150] In some possible implementations, step 2029 can be executed by an audio generation node in the engine layer calling the corresponding large audio model in the task interface layer.

[0151] The voice characteristics of the aforementioned characters may include timbre, speech rate, and volume. This application embodiment can generate voice features for different characters based on these characteristics, making the dialogue audio more suitable for the characters. The audio also includes narration audio. The execution node calls the audio model to generate audio based on the audio text, specifically including: extracting narration text from the audio text to generate narration audio according to the narration text. This achieves flexible configuration of the narration audio.

[0152] Of course, users can also trigger updates, deletions, debugging, status management, and historical execution record management of video generation workflows at the workflow platform layer.

[0153] Specifically, the first terminal receives operation information triggered by the user in the workflow platform layer, and generates a configuration adjustment request based on the operation information and sends it to the first server. The workflow orchestration service layer in the first server extracts the operation type from the operation information.

[0154] If the operation type is an update operation, the workflow orchestration service layer extracts update information from the operation information to update the video generation workflow indicated by the update information. The update information includes: the workflow identifier of the updated video generation workflow, the task identifier of the updated task, and the attributes of the updated task.

[0155] If the above operation type is a deletion operation, then the workflow orchestration service layer extracts the workflow identifier of the deleted video generation workflow from the operation information, so as to delete the video generation workflow corresponding to the workflow identifier.

[0156] If the above operation type is a debugging operation type, then the workflow orchestration service layer extracts the workflow identifier of the debugging video generation workflow from the operation information, and executes the video generation workflow corresponding to the workflow identifier for the preset debugging text. The execution process is the same as steps 201 to 202.

[0157] If the above operation type is a status management type, the workflow orchestration service layer extracts the workflow identifier of the managed video generation workflow from the operation information to query the status of the video generation workflow corresponding to that workflow identifier. The default status of the video generation workflow is "idle". When the video generation workflow is started for a target text in step 202, the workflow execution service layer sets the status of the video generation workflow to "running" and sets its status to "idle" when the video generation workflow finishes execution.

[0158] If the above operation type is an execution record management type, the workflow orchestration service layer extracts the workflow identifier of the managed video generation workflow from the operation information to query the execution record of the video generation workflow corresponding to the workflow identifier. The execution record includes: execution time, target text or target text identifier, output data of each task for each text segment of the target text, and the target video generated by the executed video generation workflow for the target text. This application embodiment can flexibly configure and manage video generation workflows, improving the flexibility and personalization of video generation workflows to meet the personalized needs of different users in different scenarios.

[0159] In some possible implementations, after step 210 and before step 202, the user may further adjust the video generation workflow to perform step 202 based on the adjusted video generation workflow. Specifically, the adjustment process includes: Step 211: Receive the adjustment request for the video generation workflow.

[0160] Step 212: Adjust the attributes of at least one task in the video generation workflow according to the adjustment request.

[0161] In some possible implementations, steps 211 to 212 may be performed by the workflow orchestration service layer.

[0162] Specifically, the adjustment request in step 211 includes adjustment operation information, which can be based on... Figure 1 The adjustment operations input in the workflow platform layer are generated. Users can input adjustment operations in the workflow platform layer. The adjustment operation information corresponding to the adjustment operation includes: the workflow identifier of the adjusted video generation workflow, the task identifier of the task, and the attributes of the adjusted task. The attributes of the adjusted task include: the execution parameters of the adjusted task, the quality detection interface of the adjusted task, and the task interface of the adjusted task.

[0163] The adjustment operation information includes the workflow identifier of the video generation workflow to be adjusted and the task identifier to be adjusted. The video generation workflow to be adjusted is determined based on its workflow identifier, and the task to be adjusted within that workflow is determined based on its task identifier, so that the attributes of the task to be adjusted are replaced with the attributes of the adjusted task.

[0164] The aforementioned adjustments are used to fine-tune the attributes of multiple tasks while maintaining the overall framework of the video generation workflow. This allows users to tweak existing video generation workflows and use them as the current workflow, effectively reducing user complexity. For example, a user can adjust the task interface of the role graph generation task in the video generation workflow and adjust the length of the target video output by the video compositing task to 30 minutes. The adjustment information includes: the video generation workflow identifier L1, the task identifier R1, the adjusted task interface J1, and the adjusted target video length of 30 minutes. The workflow orchestration service layer updates the task interface of the task corresponding to identifier R1 in the video generation workflow corresponding to L1 to J1, and sets the execution parameter "target video duration" of the video compositing task in the video generation workflow corresponding to L1 to 30 minutes. This results in a new video generation workflow.

[0165] In summary, this application embodiment achieves fully automated video generation through a video generation workflow. The workflow requires no human intervention, which helps improve video generation efficiency, reduce labor costs, facilitates large-scale video generation, enables automated quality inspection, and provides reliable quality assessment standards. Compared to existing technologies where producing a single animated series takes more than 5 days, this application embodiment can reduce production time to less than 8 hours. Furthermore, during the execution of the video generation workflow, the output data of each task is recorded to support the re-execution of partial tasks and text segments, enabling segmented debugging of the video generation workflow. Compared to re-executing the entire workflow, this application embodiment can effectively improve the speed and efficiency of re-execution.

[0166] Reference Figure 8 The diagram illustrates a block diagram of a video generation apparatus 400 provided in an embodiment of this application. The video generation apparatus 400 includes: The acquisition module 401 is used to acquire the target text and video generation workflow. The video generation workflow includes multiple tasks with dependencies, and the target text includes at least one text segment.

[0167] The execution module 402 is used to execute a video generation workflow for the target text and record the output data of the task for at least one text segment.

[0168] The determination module 403 is used to determine the re-execution text segment and the re-execution task when a segmented debugging instruction is received; the re-execution text segment includes the text segment in the target text corresponding to the target content indicated by the segmented debugging instruction; the re-execution task includes the target task indicated by the segmented debugging instruction and the downstream task of the target task in the video generation workflow.

[0169] The first execution module 404 is used to re-execute the re-execution task based on the output data of the direct dependent tasks of the re-execution task and the re-execution text segment, and obtain the target video corresponding to the target text when the video generation workflow is completed.

[0170] Reference Figure 9 As shown, the aforementioned acquisition module 401, execution module 402, determination module 403, and first execution module 404 can be configured in... Figure 1 The workflow execution service layer 105 shown is disposed in the aforementioned device 400.

[0171] Optionally, refer to Figure 9 As shown, the video generation device 400 also includes Figure 1 The workflow orchestration service layer 106 shown further includes: The configuration receiving module 405 is used to receive a workflow configuration request before acquiring the video generation workflow. The workflow configuration request is used to configure the attributes and dependencies of multiple tasks. The attributes of the tasks include at least one of the following: the task interface of the task, the quality detection interface of the task, and the execution parameters of the task interface. The task interface is used to execute the task according to the execution parameters, and the quality detection interface is used to perform quality detection on the output data of the task.

[0172] The generation module 406 is used to generate a video generation workflow based on the workflow configuration request.

[0173] Optionally, refer to Figure 9 As shown, the workflow orchestration service layer 106 also includes an adjustment module 407, used for: Before executing the video generation workflow for the target text, receive an adjustment request for the video generation workflow; adjust the attributes of at least one task in the video generation workflow according to the adjustment request.

[0174] Optionally, the execution module 402 is also used for: Record the target text identifier, the task identifier, the text segment identifier, and the task's output data for the text segment according to the corresponding relationship.

[0175] Accordingly, the first execution module 404 is also used for: Extract the re-execution text identifier from the segmented debugging instructions; determine the re-execution text segment identifier and the re-execution task identifier; for any re-execution task, based on the re-execution text identifier, the re-execution task identifier, and the re-execution text segment identifier, obtain the output data of the re-execution text segment from the output data of the directly dependent tasks of the re-execution task, and use it as the input data of the re-execution task to re-execute the re-execution task.

[0176] Optionally, the execution module 402 is also used for: Record the version information of the output data. The version information is used to distinguish multiple outputs of text segments by the task.

[0177] Accordingly, the first execution module 404 is also used for: Obtain the first version information selected from at least one version information; extract the first version information and the output data corresponding to the re-execution text segment from the output data of the direct dependent tasks of the re-execution task as the input data of the re-execution task.

[0178] Optionally, refer to Figure 9 As shown, the workflow execution service layer 105 also includes a rollback module 408, used for: Obtain the second version information; select the output data corresponding to the second version information from the output data of the last task as the target video.

[0179] Optionally, the execution module 402 is also used for: Execute independent tasks in the video generation workflow in parallel; execute dependent tasks in the video generation workflow sequentially according to their dependencies.

[0180] Optionally, the execution module 402 is further configured to: generate execution information for multiple tasks of the target text based on the target text and video generation workflow, and add the execution information of multiple tasks to the execution queue of the execution node corresponding to the task in the engine layer, so that the execution node executes the task. The execution information includes: the attributes of the task and the input data of the task. The input data of the first task is the target text, and the input data of the remaining tasks other than the first task is the output data of the tasks that directly depend on the remaining tasks.

[0181] Optionally, the video generation workflow includes: The character extraction task is used to extract character feature text from target text.

[0182] The storyboard task is used to divide the target text into multiple storyboard texts.

[0183] The task of generating cue words is used to generate character cue words based on character feature text, generate storyboard cue words based on storyboard text, and extract audio text from storyboard text.

[0184] The character illustration generation task is used to generate character illustrations based on character prompts.

[0185] The storyboard generation task is used to generate at least one storyboard based on storyboard cues, the scene style reference image corresponding to the storyboard cues, and the character image.

[0186] The audio generation task is used to generate audio from audio text.

[0187] A video clip generation task is used to generate video clips of storyboard text based on at least one storyboard diagram of the storyboard text.

[0188] The video compositing task is used to combine video clips and audio from multiple storyboard texts into a target video.

[0189] Optionally, the execution parameters for the character extraction task include: character feature type; the execution parameters for the storyboard task include: number of storyboards and length range of storyboards; the execution parameters for the cue word generation task may include: number and type of cue words; the execution parameters for the character image generation task include: clarity of the character image, size of the character image, and number of character images; the execution parameters for the storyboard image generation task include: number of storyboard images corresponding to each storyboard text, clarity of the storyboard images, and size of the storyboard images; the execution parameters for the audio generation task include: volume of the narration, speech rate of the narration, timbre of the narration, and emotion of the narration; the execution parameters for the video clip generation task include: duration of the video clip, frame rate of the video clip, and motion amplitude of adjacent frames; and the execution parameters for the video synthesis task include: duration of the target video and video format.

[0190] Optionally, the audio includes dialogue audio of at least one character. The video generation workflow further includes: a sound library generation task, which generates voice features of the characters based on the character feature text extracted by the character extraction task, and adds the voice features and the character's name from the character feature text to the character sound library. The audio includes dialogue audio of at least one character, as shown in the reference. Figure 9 As shown, the device 400 also includes an engine layer 104, which includes an execution node 1041, the execution node 1041 being further configured to: Extract at least one character's name and dialogue text from the audio text; extract the character's voice features from a character voice library based on the character's name in the audio text; generate the character's dialogue audio based on the voice features and the character's dialogue text.

[0191] Optionally, refer to Figure 9 As shown, the workflow execution service layer 105 also includes: The quality inspection module 409 is used to call the task's quality inspection interface after the task is executed to inspect the quality of the task's output data and obtain the quality inspection result.

[0192] The second execution module 410 is used to adjust the execution parameters of the task to re-execute the task if the quality detection result of the task does not meet the preset conditions of the task.

[0193] Optionally, the preset conditions for the storyboard generation task include at least one of the following: the clarity of the storyboard is greater than or equal to the preset clarity, and the matching degree between the storyboard and the storyboard prompts is greater than or equal to the preset matching degree.

[0194] The preset conditions for the character image generation task include at least one of the following: the matching degree between the character image and the character prompt is greater than or equal to the preset matching degree, and the clarity of the character image is greater than or equal to the preset clarity.

[0195] The preset conditions for the audio generation task include at least one of the following: the difference between the duration of the audio and the duration of the audio text is less than or equal to a first preset percentage, and the duration of the audio within a preset volume range is greater than or equal to a second preset percentage.

[0196] The preset conditions for the video clip generation task include at least one of the following: the inter-frame pixel difference of the video clip is greater than the preset pixel difference, and the optical flow motion of the video clip is greater than the preset optical flow motion.

[0197] Optionally, execution node 1041 is also used for: The video clips and audio clips of multiple storyboard texts are combined into a composite video according to the text segments. The composite video includes at least one video clip. The composite video of at least one text segment is combined into a target video. The at least one text segment includes: the re-executed text segment and the non-re-executed text segment other than the re-executed text segment.

[0198] Optionally, execution node 1041 is also used for: Add silent audio before and / or after the audio of the storyboard text to obtain the storyboard audio; combine video clips of multiple storyboard texts and the storyboard audio of multiple storyboard texts into a composite video of text segments.

[0199] Optionally, execution node 1041 is also used for: If the duration difference between the audio and video clips of the storyboard text is less than or equal to the preset duration difference, the video clips and audio of multiple storyboard texts will be combined into the target video.

[0200] In summary, this application achieves fully automated video generation through a video generation workflow. The workflow requires no human intervention, significantly improving production efficiency. Compared to existing technologies where producing a single animated series takes over 5 days, this application's embodiment can reduce production time to less than 8 hours. Furthermore, during the execution of the video generation workflow, the output data of each task is recorded to support the re-execution of partial tasks and text segments, enabling segmented debugging of the video generation workflow. Compared to a complete re-execution, this application's embodiment effectively improves the speed and efficiency of re-execution.

[0201] This application also provides a non-volatile readable storage medium storing one or more modules (programs). When these modules are applied to a device, they enable the device to execute the instructions for the method steps in this application.

[0202] This application provides one or more machine-readable media storing instructions that, when executed by one or more processors, cause an electronic device to perform one or more methods as described in the above embodiments. In this application, the electronic device includes various types of devices such as terminal devices and servers (clusters).

[0203] The embodiments of this disclosure can be implemented as an apparatus configured as desired using any suitable hardware, firmware, software, or any combination thereof, including electronic devices such as terminal devices, servers (clusters), etc. Figure 10 An exemplary apparatus 1000 is schematically shown that can be used to implement various embodiments of the present application.

[0204] In one embodiment, Figure 10 An exemplary device 1000 is shown, which includes one or more processors 1002, a control module (chipset) 1004 coupled to at least one of the processors 1002, a memory 1006 coupled to the control module 1004, a non-volatile memory (NVM) / storage device 1008 coupled to the control module 1004, one or more input / output devices 1010 coupled to the control module 1004, and a network interface 1012 coupled to the control module 1004.

[0205] Processor 1002 may include one or more single-core or multi-core processors, and processor 1002 may include any combination of general-purpose processors or special-purpose processors (e.g., graphics processors, application processors, baseband processors, etc.). In some embodiments, device 1000 can serve as a terminal device, server (cluster), or other device in the embodiments of this application.

[0206] In some embodiments, apparatus 1000 may include one or more computer-readable media (e.g., memory 1006 or NVM / storage device 1008) having instructions 1014 and one or more processors 1002 that are combined with the one or more computer-readable media and configured to execute the instructions 1014 to implement the module and thus perform the actions in this disclosure.

[0207] In one embodiment, the control module 1004 may include any suitable interface controller to provide any suitable interface to at least one of the processors 1002 and / or any suitable device or component communicating with the control module 1004.

[0208] The control module 1004 may include a memory controller module to provide an interface to the memory 1006. The memory controller module may be a hardware module, a software module, and / or a firmware module.

[0209] Memory 1006 may be used, for example, to load and store data and / or instructions 1014 for device 1000. In one embodiment, memory 1006 may include any suitable volatile memory, such as suitable dynamic random access memory (DRAM). In some embodiments, memory 1006 may include double data rate type quad synchronous dynamic random access memory.

[0210] In one embodiment, the control module 1004 may include one or more input / output controllers to provide interfaces to the NVM / storage device 1008 and (one or more) input / output devices 1010.

[0211] For example, NVM / storage device 1008 may be used to store data and / or instructions 1014. NVM / storage device 1008 may include any suitable non-volatile memory (e.g., flash memory) and / or may include any suitable (one or more) non-volatile storage devices (e.g., one or more hard disk drives (HDDs), one or more optical disc drives (CDs), and / or one or more digital video disc drives (DVDs)).

[0212] NVM / storage device 1008 may include storage resources that are physically part of a device on which device 1000 is mounted, or that are accessible to the device but do not necessarily have to be part of the device. For example, NVM / storage device 1008 may be accessed via a network via one or more input / output devices 1010.

[0213] One or more input / output devices 1010 may provide an interface for device 1000 to communicate with any other suitable device. Input / output devices 1010 may include communication components, audio components, sensor components, etc. Network interface 1012 may provide an interface for device 1000 to communicate via one or more networks. Device 1000 may wirelessly communicate with one or more components of a wireless network according to any of one or more wireless network standards and / or protocols, such as accessing wireless networks based on communication standards, such as WiFi, 2G, 3G, 4G, 5G, etc., or combinations thereof.

[0214] In one embodiment, at least one of the processors 1002 may be logically packaged with one or more controllers (e.g., memory controller modules) of the control module 1004. In one embodiment, at least one of the processors 1002 may be logically packaged with one or more controllers of the control module 1004 to form a system-in-package (SoC). In one embodiment, at least one of the processors 1002 may be integrated with the logic of one or more controllers of the control module 1004 on the same die. In one embodiment, at least one of the processors 1002 may be integrated with the logic of one or more controllers of the control module 1004 on the same die to form a system-on-chip (SoC).

[0215] In various embodiments, device 1000 may be, but is not limited to, a server, desktop computing device, or mobile computing device (e.g., laptop computing device, handheld computing device, tablet computer, netbook, etc.). In various embodiments, device 1000 may have more or fewer components and / or different architectures. For example, in some embodiments, device 1000 includes one or more cameras, a keyboard, a display (including a touchscreen display), a non-volatile memory port, multiple antennas, a graphics chip, an application-specific integrated circuit (ASIC), and a speaker.

[0216] The detection device can use a main control chip as a processor or control module, and sensor data, position information, etc. can be stored in a memory or NVM / storage device. The sensor group can be used as an input / output device, and the communication interface can include a network interface.

[0217] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0218] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0219] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, create means for implementing the functions specified in one or more blocks of the flowchart illustrations and / or one or more blocks of the block diagrams.

[0220] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowcharts and / or one or more blocks of a block diagram.

[0221] These computer program instructions may also be loaded onto a computer or other programmable data processing terminal equipment to cause a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable terminal equipment, provide steps for implementing the functions specified in one or more flowcharts and / or one or more blocks of a block diagram.

[0222] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0223] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes the element.

[0224] The above provides a detailed description of a video generation method, apparatus, electronic device, and machine-readable medium provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. Under different circumstances, those skilled in the art will find changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the text of this specification should not be construed as a limitation of this application.

Claims

1. A video generation method, characterized in that, The method includes: Obtain the target text and video generation workflow, wherein the video generation workflow includes multiple tasks with dependencies, and the target text includes at least one text segment; The video generation workflow is executed for the target text, and the output data of the task for the at least one text segment is recorded; Upon receiving a segmented debugging instruction, the re-execution text segment and the re-execution task are determined; the re-execution text segment includes the text segment corresponding to the target content indicated by the segmented debugging instruction in the target text; the re-execution task includes the target task indicated by the segmented debugging instruction and the downstream task of the target task in the video generation workflow; Based on the output data of the direct dependent tasks of the re-execution task and the re-execution text segment, the re-execution task is re-executed, and the target video corresponding to the target text is obtained when the video generation workflow is completed.

2. The method according to claim 1, characterized in that, Before obtaining the video generation workflow, it also includes: A workflow configuration request is received. The workflow configuration request is used to configure the attributes of the multiple tasks and the dependencies. The attributes of the tasks include at least one of the following: the task interface of the task, the quality detection interface of the task, and the execution parameters of the task interface. The task interface is used to execute the task according to the execution parameters, and the quality detection interface is used to perform quality detection on the output data of the task. The video generation workflow is generated based on the workflow configuration request.

3. The method according to claim 2, characterized in that, Before executing the video generation workflow for the target text, the process further includes: Receive adjustment requests for the video generation workflow; The attributes of at least one task in the video generation workflow are adjusted according to the adjustment request.

4. The method according to claim 1, characterized in that, Record the output data of the task for the at least one text segment, including: The text identifier of the target text, the task identifier of the task, the text segment identifier of the text segment, and the output data of the task to the text segment are recorded according to the corresponding relationship; Determine the text segment to be re-executed and the re-execution task, including: Extract the re-execution text identifier from the segmented debugging instructions; Determine the re-execution text segment identifier and the re-execution task identifier; Based on the output data of the tasks directly dependent on the re-execution task and the re-execution text segment, the re-execution task is re-executed, including: For any of the re-execution tasks, based on the re-execution text identifier, the re-execution task identifier, and the re-execution text segment identifier, the output data of the re-execution text segment is obtained from the output data of the directly dependent tasks of the re-execution task and used as the input data of the re-execution task to re-execute the re-execution task.

5. The method according to claim 4, characterized in that, Also includes: Record the version information of the output data, which is used to distinguish the multiple output data of the text segment by the task; The output data of the re-executed text segment is obtained from the output data of the tasks directly dependent on the re-execution task, and used as the input data of the re-execution task, including: Obtain first version information selected from at least one of the version information; Extract the first version information and the output data corresponding to the re-execution text segment from the output data of the direct dependent tasks of the re-execution task as the input data of the re-execution task.

6. The method according to any one of claims 1 to 5, characterized in that, The video generation workflow is executed for the target text, including: Based on the target text and the video generation workflow, execution information for multiple tasks related to the target text is generated, and the execution information for these multiple tasks is added to the execution queue of the execution node corresponding to the task in the engine layer, so that the execution node executes the task. The execution information includes: the attributes of the task and the input data of the task. The input data for the first task is the target text, and the input data for the remaining tasks other than the first task is the output data of the directly dependent tasks of the remaining tasks.

7. The method according to claim 2, characterized in that, The video generation workflow includes: A character extraction task is used to extract character feature text from the target text. The storyboard task is used to divide the target text into multiple storyboard texts. The cue word generation task is used to generate character cue words based on the character feature text, generate storyboard cue words based on the storyboard text, and extract audio text from the storyboard text; The character image generation task is used to generate character images based on the character prompts. The storyboard generation task is used to generate at least one storyboard based on the storyboard prompt, the scene style reference image corresponding to the storyboard prompt, and the character image. An audio generation task is used to generate audio based on the audio text. A video clip generation task is used to generate a video clip of the storyboard text based on at least one storyboard diagram of the storyboard text. The video synthesis task is used to synthesize the video clips of the multiple storyboard texts and the audio into the target video.

8. The method according to claim 7, characterized in that, The execution parameters for the character extraction task include: character feature type; The execution parameters for the storyboard task include the number of storyboards and the length range of the storyboards; The execution parameters for the prompt word generation task include: the number and type of prompt words; The execution parameters for the character image generation task include: the clarity of the character image, the size of the character image, and the number of character images. The execution parameters for the storyboard generation task include: the number of storyboards corresponding to each storyboard text, the clarity of the storyboard, and the size of the storyboard. The execution parameters for the audio generation task include: the volume of the narration, the speaking speed of the narration, the timbre of the narration, and the emotion of the narration. The execution parameters for the video segment generation task include: the duration of the video segment, the frame rate of the video segment, and the motion amplitude of adjacent frames; The execution parameters of the video synthesis task include: the duration and format of the target video.

9. The method according to claim 7, characterized in that, The audio includes dialogue audio of at least one character. The video generation workflow further includes: a sound library generation task, which generates voice features of the character based on the character feature text extracted by the character extraction task, and adds the voice features and the character's name from the character feature text to the character sound library. The audio includes dialogue audio of at least one character, and generating audio based on the audio text includes: Extract the name of at least one character and the dialogue text of that character from the audio text; Extract the voice features of the character from the character voice library based on the character's name in the audio text; The dialogue audio of the character is generated based on the voice features and the character's dialogue text.

10. The method according to claim 7, characterized in that, Also includes: After the task is executed, the quality detection interface of the task is called to detect the quality of the output data of the task and obtain the quality detection result. If the quality inspection result of the task does not meet the preset conditions of the task, the execution parameters of the task are adjusted to re-execute the task.

11. The method according to claim 10, characterized in that, The preset conditions for the storyboard generation task include at least one of the following: the clarity of the storyboard is greater than or equal to a preset clarity, and the matching degree between the storyboard and the storyboard prompt is greater than or equal to a preset matching degree. The preset conditions for the character image generation task include at least one of the following: the matching degree between the character image and the character prompt is greater than or equal to the preset matching degree, and the clarity of the character image is greater than or equal to the preset clarity. The preset conditions for the audio generation task include at least one of the following: the difference between the duration of the audio and the reading duration of the audio text is less than or equal to a first preset percentage, and the duration of the audio within a preset volume range is greater than or equal to a second preset percentage. The preset conditions for the video segment generation task include at least one of the following: the inter-frame pixel difference of the video segment is greater than the preset pixel difference, and the optical flow motion of the video segment is greater than the preset optical flow motion.

12. The method according to claim 7, characterized in that, The process of combining the video clips of the multiple storyboard texts and the audio into the target video includes: Add silent audio before and / or after the audio of the storyboard text to obtain the storyboard audio; The video clips and audio clips of the multiple storyboard texts are combined into a composite video based on the text segments. The synthesized video is composed of at least one text segment, wherein the at least one text segment includes: the re-executed text segment and non-re-executed text segments other than the re-executed text segment.

13. A video generation apparatus, characterized in that, The device includes: The acquisition module is used to acquire the target text and video generation workflow, wherein the video generation workflow includes multiple tasks with dependencies, and the target text includes at least one text segment; An execution module is used to execute the video generation workflow for the target text and record the output data of the task for the at least one text segment; The determination module is used to determine the re-execution text segment and the re-execution task upon receiving a segmented debugging instruction; the re-execution text segment includes the text segment corresponding to the target content indicated by the segmented debugging instruction in the target text; the re-execution task includes the target task indicated by the segmented debugging instruction and the downstream task of the target task in the video generation workflow; The first execution module is used to re-execute the re-execution task based on the output data of the direct dependent tasks of the re-execution task and the re-execution text segment, and obtain the target video corresponding to the target text when the video generation workflow is completed.

14. An electronic device, characterized in that, include: processor; and A memory having executable code stored thereon, which, when executed, causes the processor to perform the method as described in any one of claims 1 to 12.

15. One or more machine-readable media, characterized in that, It stores executable code that, when executed, causes the processor to perform the method as described in any one of claims 1 to 12.