A method, system, and computer device for generating text-based videos.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- GUANGDONG HENGQIN SHUSHUSHUO STORY INFORMATION TECH CO LTD
- Filing Date
- 2025-12-26
- Publication Date
- 2026-05-26
Smart Images
Figure CN122093633A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of video processing technology, and in particular to a method, system and computer device for generating text-based videos. Background Technology
[0002] Text-based video generation technology, as an important branch of multimodal generation technology, has become a hot research topic due to its core advantage of directly converting text descriptions into continuous video content. It has broad application prospects in multiple fields such as content creation, media dissemination, and digital entertainment.
[0003] Currently, textual video generation technology mainly relies on large language models and multimodal models. On the one hand, the performance of the large language models and multimodal models that early textual videos relied on was limited, making it difficult to achieve satisfactory results in a single generation. Satisfactory results had to be obtained through a "gambling" method, with the generation effect depending on luck, resulting in extremely poor stability and efficiency. On the other hand, the existing textual video generation workflow design is rudimentary. Early workflows were mostly unidirectional and fixed processes, generating videos through storyboard images and splicing segments, resulting in fragmented images, low quality, and difficulty in maintaining and optimizing the workflow. In addition, existing textual video generation methods usually adopt a workflow design with fixed nodes and a unidirectional, one-time output design. This architecture lacks good evaluation nodes in the textual video generation process, and cannot automatically provide feedback and optimize for re-output, making it difficult to form a closed-loop textual video generation. At the same time, the final video merging process in the image-to-video stage lacks effective merging logic, usually merging sequentially, which is not intelligent enough and leads to poor quality of textual video generation.
[0004] Therefore, how to provide a closed-loop evaluation node for text-based video generation and improve the quality of text-based video generation has become an urgent technical problem to be solved. Summary of the Invention
[0005] To address the problems of existing technologies, such as the lack of evaluation nodes, difficulty in forming a closed loop for text-based video generation, and poor quality of text-based video generation, this invention proposes a text-based video generation method, system, and computer equipment that can effectively provide evaluation nodes to form a closed loop for text-based video generation, thereby improving the quality of text-based video generation.
[0006] To achieve the above-mentioned technical effects, the technical solution of the present invention is as follows: A method for generating text-based videos includes the following steps: S1. Obtain user input parameters, and generate anchor prompt words and anchor role diagrams based on the user input parameters; S2. Evaluate the anchor point role diagram until the anchor point role diagram passes the evaluation, then proceed to S3; S3. Extract information from the anchor point prompts to obtain the storyboard; S4. Evaluate the storyboard until the storyboard passes the evaluation, then proceed to S5; S5. Based on the storyboard script, generate audio and storyboard video respectively; S6. Evaluate the storyboard video until the storyboard video evaluation passes, then proceed to S7; S7. Post-process the storyboard video to obtain a post-processed video, and combine the post-processed video with the audio to generate the target video.
[0007] Preferably, the step of generating anchor prompts and anchor role diagrams based on the user input parameters includes: S11. Determine whether the user input parameters contain a custom image. If yes, proceed to S12; otherwise, proceed to S13. S12. Input the user input parameters into the preset large language model, and generate anchor prompt words from the output image of the large language model, and skip generating the anchor role diagram; S13. Input the user input parameters into a preset large language model, and the large language model outputs an image to generate anchor point prompts. Input the image-generated anchor point prompts into a preset visual language model, and the visual language model outputs an anchor point role diagram.
[0008] Preferably, the evaluation of the anchor point role diagram until the anchor point role diagram passes the evaluation includes: The user input parameters, the anchor prompt words, and the anchor role diagram are input into the visual language model. The visual language model analyzes whether the core elements in the anchor role diagram match the anchor prompt words. If they match, the anchor role diagram passes the evaluation; if not, the anchor role diagram fails the evaluation. The reason for the failure is input into the large language model, which then regenerates the anchor prompt words.
[0009] Preferably, the step of extracting information from the anchor cue words to obtain a storyboard includes: inputting the anchor cue words into the large language model, extracting the main information of the anchor cue words from the large language model, and generating a structured storyboard based on the main information.
[0010] Preferably, the step of evaluating the storyboard until the storyboard passes evaluation includes: S41. Encapsulate the storyboard script according to a preset format to obtain a storyboard preset format script; S42. Input the storyboard preset format script into the large language model. Use the large language model to analyze whether there is a conflict between the content outline and the visual style of the storyboard preset format script. If so, the storyboard script fails the evaluation. Output the storyboard index and the reason for the conflict. Input the storyboard index, the reason for the conflict, and the anchor point prompt into the large language model. The large language model regenerates the storyboard script. Continue until the regenerated storyboard script fails the evaluation several times. Then, directly select the storyboard script to participate in the last evaluation and execute S5. If not, the storyboard script passes the evaluation and execute S5.
[0011] Preferably, the step of generating audio and storyboard video based on the storyboard script includes: S51. Input the storyboard into a preset text-to-speech model, and have the text-to-speech model output the audio. S52. Fill the storyboard script, the user input parameters and the anchor character diagram into the storyboard image generation prompt, input the storyboard image generation prompt into the large language model, and output the storyboard image from the large language model; S53. Input the storyboard images and video generation prompts into a preset video generation model, and output the storyboard video from the video generation model.
[0012] Preferably, the step of evaluating the storyboard video until the storyboard video evaluation passes includes: S61. Input the storyboard video and the video-generated prompts into the large language model. The large language model performs a binary evaluation based on three dimensions: subject consistency, story integrity, and video performance, and outputs a binary evaluation result, which is 0 or 1. S62. Sum the results of the two-factor evaluation to obtain the total evaluation score; S63. Determine whether the total evaluation score is equal to 3. If yes, the storyboard video evaluation is passed; otherwise, the storyboard video evaluation is not passed. Preferably, the post-processing of the storyboard video to obtain a post-processed video includes: S71. Use the large language model to sort the storyboard videos to obtain the sorted storyboard videos; S72. Extract the last frame of the previous storyboard video and the first frame of the next storyboard video from the storyboard video sequence; S73. Using a preset image feature extraction model, extract the first embedding vector of the last frame of the previous storyboard video and the second embedding vector of the first frame of the next storyboard video, and calculate the cosine similarity between the first embedding vector and the second embedding vector. S74. Compare the cosine similarity with the transition threshold to obtain the comparison result, and select different transitions to generate a transition video based on the comparison result; S75. Merge the storyboard video and the transition video to obtain a merged video, and then splice the merged video with the audio to obtain the target video.
[0013] This invention also proposes a text-based video generation system, comprising: The first generation module is used to obtain user input parameters and generate anchor prompt words and anchor role diagrams based on the user input parameters. The first evaluation module is used to evaluate the anchor point role diagram until the anchor point role diagram passes the evaluation; The information extraction module is used to extract information from the anchor point prompts after the anchor point character diagram has passed the evaluation, and to obtain the storyboard script. The second evaluation module is used to evaluate the storyboard until the storyboard passes the evaluation. The second generation module is used to generate audio and storyboard video based on the storyboard after the storyboard evaluation is passed. The third evaluation module is used to evaluate the storyboard video until the storyboard video passes the evaluation; The third generation module is used to post-process the storyboard video to obtain a post-processed video, and then combine the post-processed video with the audio to generate the target video.
[0014] The present invention also proposes a computer device, comprising: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other through the communication bus; The memory is used to store at least one executable instruction that causes the processor to perform the operation of the text-based video generation method as described above.
[0015] Compared with the prior art, the beneficial effects of the technical solution of the present invention are: This invention proposes a text-based video generation method, system, and computer equipment. First, anchor point prompts and anchor point character diagrams are generated based on user input parameters. Then, the anchor point character diagrams are evaluated to ensure accurate character portrayals, laying a solid foundation for the video. Next, anchor point prompt information is extracted to obtain a storyboard, which is then evaluated to ensure clear script logic and plot coherence. Based on the evaluated script, audio and storyboard video are generated, and the storyboard video is evaluated again to improve image quality and motion smoothness. Finally, the target video is synthesized through post-processing and audio synthesis, thereby improving the quality of the text-based video generation. This invention also constructs a closed-loop control system by setting evaluation nodes in the three key stages of anchor point character diagrams, storyboard scripts, and storyboard video, effectively establishing a closed-loop link for text-based video generation and ensuring the controllability and traceability of the generation process. Attached Figure Description
[0016] Figure 1 This is a flowchart illustrating a text-based video generation method proposed in an embodiment of the present invention. Figure 2 This is a schematic diagram of the structure based on the LangGraph framework proposed in the embodiments of the present invention; Figure 3 This diagram illustrates the structure of the LLM module proposed in this embodiment of the invention. Figure 4 This diagram illustrates the structure of the VLM module proposed in this embodiment of the invention. Figure 5 This diagram illustrates the structure of the TTS module proposed in this embodiment of the invention. Figure 6 This diagram illustrates the structure of the I2V module proposed in this embodiment of the invention. Figure 7 This diagram illustrates the structure of the video processing module proposed in this embodiment of the invention. Figure 8 This diagram illustrates the structural block diagram of a text-based video generation system proposed in this embodiment of the invention. Figure 9 This is a structural block diagram of a computer device proposed in an embodiment of the present invention.
[0017] 91. Processor; 92. Memory; 93. Communication interface; 94. Communication bus; 95. Executable instructions. Detailed Implementation
[0018] The accompanying drawings are for illustrative purposes only and should not be construed as limiting the scope of this patent. It is understandable to those skilled in the art that some well-known details may be omitted from the accompanying drawings; The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.
[0019] Example 1 See Figure 1 This embodiment proposes a text-based video generation method, including the following steps: S1. Obtain user input parameters, and generate anchor prompt words and anchor role diagrams based on the user input parameters; In S1, the user input parameters include the user-input topic and other input parameters `user_params`, such as overall style, characters, and items. Based on the user input parameters, generating anchor prompts and anchor character diagrams includes: S11. Determine whether the user input parameters contain a custom image. If yes, proceed to S12; otherwise, proceed to S13. S12. Input the user input parameters into the preset large language model, and generate anchor prompt words from the output image of the large language model, and skip generating the anchor role diagram; In S12, if the user directly passes in a custom image through the anchor_img field of the other input parameter user_params in the user input parameters, then the generation of the anchor point role graph is skipped, and the anchor point prompt words are generated based on the user-provided image and other information through the preset large language model LLM.
[0020] S13. Input the user input parameters into a preset large language model, and the large language model outputs an image to generate anchor point prompts. Input the image-generated anchor point prompts into a preset visual language model, and the visual language model outputs an anchor point role diagram.
[0021] S2. Evaluate the anchor point role diagram until the anchor point role diagram passes the evaluation, then proceed to S3; In S2, the evaluation of the anchor point role diagram until the anchor point role diagram passes the evaluation includes: The user input parameters, the anchor prompt words, and the anchor role diagram are input into the visual language model. The visual language model analyzes whether the core elements in the anchor role diagram match the anchor prompt words. If they match, the anchor role diagram passes the evaluation; if not, the anchor role diagram fails the evaluation. The reason for the failure is input into the large language model, which then regenerates the anchor prompt words.
[0022] Here, the anchor prompts and anchor images generated by S1 are crucial to the overall process and affect the overall generation effect. Therefore, it is necessary to evaluate the effect of the anchor role image. If the user provides a custom image, the evaluation step is skipped: the anchor role image is directly re-inputted through the visual language model, then analyzed and evaluated. The evaluation results and the local save path of the anchor role image are returned to the log and exposed on the terminal. If the evaluation passes, proceed directly to the next node. If it fails, the evaluation reason is passed to the Large Language Model (LLM) to regenerate the anchor prompts.
[0023] S3. Extract information from the anchor point prompts to obtain the storyboard; In S3, the step of extracting information from the anchor cue words to obtain a storyboard includes: inputting the anchor cue words into the large language model, extracting the main information of the anchor cue words by the large language model, such as characters, items, art style, theme, actions, etc., and generating a structured storyboard based on the main information. Each storyboard includes a content outline, character actions and expressions, camera movement, background, visual style, and storyboard duration information. S4. Evaluate the storyboard until the storyboard passes the evaluation, then proceed to S5; In S4, evaluating the storyboard until the storyboard passes evaluation includes: S41. Encapsulate the storyboard script according to a preset format to obtain a storyboard preset format script; S42. Input the storyboard preset format script into the large language model. Use the large language model to analyze whether there is a conflict between the content outline and the visual style of the storyboard preset format script. If so, the storyboard script fails the evaluation. Output the storyboard index and the reason for the conflict. Input the storyboard index, the reason for the conflict, and the anchor point prompt into the large language model. The large language model regenerates the storyboard script. Continue until the regenerated storyboard script fails the evaluation several times. Then, directly select the storyboard script to participate in the last evaluation and execute S5. If not, the storyboard script passes the evaluation and execute S5.
[0024] Here, the evaluation of the storyboard script involves using the feedback logic of each storyboard...<storyboard_n> content< / storyboard_n> This XML preset format is used to distinguish between different scenes. Then, the Large Language Model (LLM) is called to analyze whether there are any conflicts between the content outlines and visual styles of different scenes. Then, the scenes with the corresponding indexes are regenerated, and each scene can be retried up to 3 times.
[0025] S5. Based on the storyboard script, generate audio and storyboard video respectively; In S5, the generation of audio and storyboard video based on the storyboard script includes: S51. Input the storyboard into a preset text-to-speech model, and have the text-to-speech model output the audio. In S51, this step is executed by the audio stream node, which controls whether to execute it through the `audio_gen` field of `user_params`, defaulting to `False`. This node calls the text-to-speech model in the TTS module, generates audio based on the content outline and scene duration of the storyboard script, and returns the local save path `audio_path`.
[0026] S52. Fill the storyboard script, the user input parameters and the anchor character diagram into the storyboard image generation prompt, input the storyboard image generation prompt into the large language model, and output the storyboard image from the large language model; 2): Fill the storyboard with the topic text, storyboard, anchor style prompt, and anchor character image into the storyboard image generation prompt. Based on this prompt and the VLM module, generate corresponding images for each storyboard. Then, pass the image, storyboard outline, and duration to the Sora2 interface to generate video clips.
[0027] S53. Input the storyboard image and video generation prompts into a preset video generation model, and output the storyboard video from the video generation model; wherein the video generation model is accessed through the interface of sora2, and the outline and duration of the storyboard image and video generation prompts are passed to the interface of sora2. S6. Evaluate the storyboard video until the storyboard video evaluation passes, then proceed to S7; In S6, evaluating the storyboard video until the storyboard video evaluation passes includes: S61. Input the storyboard video and the video-generated prompts into the large language model. The large language model performs a binary evaluation based on three dimensions: subject consistency, story integrity, and video performance, and outputs a binary evaluation result, which is 0 or 1. S62. Sum the results of the two-factor evaluation to obtain the total evaluation score; S63. Determine whether the total evaluation score is equal to 3. If yes, the storyboard video evaluation is passed; otherwise, the storyboard video evaluation is not passed. Here, after the video is generated, the corresponding storyboard image, generated prompts, and video prompts are all fed into the Visual Language Model (VLM). If the evaluation passes, the process continues to the next node; if the evaluation fails, the retry node is determined based on the analysis results to be one of the following: storyboard script generation node, storyboard image generation node, video prompt generation node, or video generation node. The reason for the analysis is added as a new input to the corresponding node, and the maximum number of retries is 5.
[0028] For video-generated prompts and storyboard images, the storyboard images, storyboard text, and video-generated prompts are fed into the Visual Language Model (VLM). Then, based on the video-generated prompt evaluation metrics, such as prompt subject element evaluation, prompt content completeness evaluation, and prompt redundancy evaluation, the VLM is judged to have any problems. A score of 3 is required to pass. Similarly, based on the storyboard image evaluation metrics, such as storyboard image subject consistency, storyboard image completeness evaluation, and storyboard image performance evaluation, the VLM is judged to have any problems.
[0029] The above two evaluation categories will be returned to the corresponding node for regeneration based on whether they pass or fail. If there are problems in both categories, the storyboard image generation node will be returned directly; if there are no problems, the "storyboard script" will be evaluated, which will also be a passing score of 3.
[0030] For storyboards, the storyboard and anchor cue words are fed into the Large Language Model (LLM). Then, the script is evaluated based on three binary evaluation metrics: script consistency, script story integrity, and script redundancy. A score of 3 is required to pass. If the script fails, the corresponding storyboard is regenerated based on the feedback information. If the script passes, the overall process is normal, and the video post-processing and compositing nodes are performed.
[0031] S7. Post-process the storyboard video to obtain a post-processed video, and combine the post-processed video with the audio to generate the target video.
[0032] In S7, the post-processing of the storyboard video to obtain a post-processed video includes: S71. Use the large language model to sort the storyboard videos to obtain the sorted storyboard videos; In S71, storyboard sorting refers to extracting the outline of each storyboard and using...<storyboard_n> content< / storyboard_n> This XML format differentiation, combined with the user's thematic text input, uses a Large Language Model (LLM) to sort the storyboards, forming a complete and fluent story logic and content logic.
[0033] S72. Extract the last frame of the previous storyboard video and the first frame of the next storyboard video from the storyboard video sequence; S73. Using the preset image feature extraction model ImageEmbedding, extract the first embedding vector emb1 of the last frame of the previous storyboard video and the second embedding vector emb2 of the first frame of the next storyboard video, respectively, and calculate the cosine similarity score between the first embedding vector and the second embedding vector; wherein the formula for calculating the cosine similarity score is as follows: score = util.cos_sim(emb1, emb2) Where util.cos_sim() represents the cosine similarity function; S74. Compare the cosine similarity with the transition threshold to obtain the comparison result, and select different transitions to generate a transition video based on the comparison result; that is, select different transitions based on the comparison between the score and the transition threshold score_threshold.
[0034] Here, the step of selecting different transitions to generate a transition video based on the comparison results includes: If the cosine similarity exceeds the transition threshold (i.e., the two shots are relatively similar with minimal visual jumps), a third-party package is used to apply fade-out and fade-in effects to the last 0.5 seconds of the first shot and the first 0.5 seconds of the second shot, respectively. Then, the 0.1-second segments of the two shots are dissolved to achieve a smooth transition.
[0035] If the cosine similarity is below the transition threshold (i.e., the two shots have low similarity and a large scene jump), directly input the two captured frame images to generate the transition video. If the image-to-video interface does not support inputting two images simultaneously, the two frames need to be horizontally stitched together. Generate a left-to-right transition based on prompts.
[0036] S75. Merge the storyboard video and the transition video to obtain a merged video. Then, splice the merged video with the audio to obtain the target video. After merging, overlay the audio text onto the corresponding position in the target video using a third-party package.
[0037] In this embodiment, anchor prompts and anchor character diagrams are first generated based on user input parameters. The anchor character diagrams are then evaluated to ensure accurate character portrayals, laying a solid foundation for the video. Next, anchor prompt information is extracted to obtain a storyboard, which is then evaluated to ensure clear script logic and coherent plot. Based on the evaluated script, audio and storyboard video are generated, and the storyboard video is evaluated again to improve image quality and motion smoothness. Finally, the target video is synthesized through post-processing and audio, thereby improving the quality of the generated text-based video. Furthermore, this invention establishes a closed-loop control system by setting evaluation nodes in the three key stages of anchor character diagrams, storyboard scripts, and storyboard video, effectively connecting the closed-loop chain of text-based video generation and ensuring the controllability and traceability of the generation process.
[0038] This embodiment, based on the text-based video generation method proposed above, also provides a text-based video architecture based on the LangGraph framework. Utilizing a large language model and a multimodal model, it can customize multi-dimensional input information, dynamically plan execution nodes, and construct a closed-loop agent with multiple evaluation stages. The overall architecture is as follows: Figure 2-7 As shown, it mainly includes the following modules: (1) Global State Layer of Graph Global parameter definition: The parameter types, descriptions, initial value definitions, and global state of the graph for different nodes and overall input and output variables.
[0039] (2) Graph functional module layer The LLM module creates the LLM_Server class, which can initialize a large language model (LLM) based on API calls. It can expand user input, reconstructing single topic text and style information into high-quality image prompts that conform to the input style of the image generation model. Furthermore, it can generate structured storyboard information based on topic information and image prompts.
[0040] The TTS module creates an ALM_Server class, which can initialize a text-to-speech (TTS) model based on the API call interface, and can generate artificially synthesized audio from the text information in the storyboard generated by the LLM module.
[0041] VLM module: Creates the VLM_Server class, which can initialize the visual language model based on the API call and generate anchor point images and storyboard images based on prompt words; The I2V module creates an I2V_Server class, which calls the sora2 interface to access the video generation model and generates video based on the text and images of each scene. Since the sora2 generated results include audio, users can choose between sora2 or TTS-generated audio based on parameters; the default is the sora2 audio track. Video processing module: Creates the Editor_Server class, which is responsible for coordinating the splicing between different scenes, as well as processing functions such as transition effects, subtitles, and editing between scenes.
[0042] (3) Graph execution node layer Node0_init: Receives initial parameters from the user through the Large Language Model (LLM) and the Visual Language Model (VLM), and is responsible for copy optimization and image anchor point generation. Subsequent generated content is evaluated and optimized based on these initial anchor points.
[0043] node1_script: Generates structured storyboard information using a large language model (LLM).
[0044] node2_audio: Generates synthesized audio using a text-to-speech (TTS) model.
[0045] node2_visual: Generates corresponding images based on storyboard information using the Visual Language Model (VLM) and the I2V module, and then generates video clips from each storyboard image.
[0046] node3_merge: This module stitches together the generated storyboard videos and adds transitions, subtitles, and other content.
[0047] General tool module Prompt layer: Centralized prompt management facilitates rapid iteration and optimization; General Tool Layer: Manages tools across functional modules and is responsible for implementing general-purpose Tools classes.
[0048] Log layer: Responsible for implementing log recording functions to facilitate data traceability and operation log recording.
[0049] (4) Graph Run and Test Layer Graph Assembly: Initialize the graph, add all execution nodes, define ordinary edges and conditional edges, build the workflow architecture and compile.
[0050] Sample testing: 1. The LangSmith module enables visualization of node parameters and targeted optimization. 2. Debugging is performed using breakpoints in the IDE software.
[0051] This embodiment proposes a more intelligent, closed-loop agent based on the LangGraph framework, capable of autonomously selecting node paths and providing feedback on output. Compared with existing technologies, the LangGraph-based agent proposed in this embodiment has the following advantages. 1. Current video editing workflows typically employ fixed-node designs and unidirectional, one-time output, often resulting in poor quality. This invention utilizes the LangGraph framework, leveraging the analytical capabilities of LLM or multimodal models, to autonomously determine the next execution path based on current node information, dynamically planning subsequent execution paths. Furthermore, this framework incorporates evaluation nodes after some node outputs, allowing for the addition of feedback information to regenerate higher-quality results for outputs below a certain quality threshold.
[0052] 2. Current video workflows often result in massive amounts of code that are difficult to customize in certain processes, requiring continuous expansion of functional boundaries. However, this invention, based on the LangGraph framework, designs different functional modules as sub-Agents. Each sub-Agent contains multiple specific execution nodes, resulting in a clear and easily manageable structure that facilitates expansion and offers a more flexible architecture.
[0053] 3. Current text-based video workflows lack dynamic node planning, validation, reflection, and retries, as well as a well-structured Agent framework, leading to redundancy and maintenance difficulties. This invention constructs a more intelligent Agent based on the LangGraph framework. This Agent can plan different implementation paths based on input information and utilize called model APIs or locally deployed model interfaces based on Vllm or ComfyUI frameworks to form a closed-loop text-based video workflow.
[0054] This embodiment provides a closed-loop agent architecture that includes AI dynamic planning nodes and evaluation and reflection, enabling a high-quality generation process that can generate images from text, videos from images, and provide additional supplementary information. The different functional modules are clearly planned, facilitating functional expansion and maintenance, and allowing for timely feedback and optimization of node output quality, thereby solving the following technical problems: 1. The architecture is chaotic, with different modules and functions mixed together, making it difficult to maintain and develop and optimize new features. 2. The framework lacks effective evaluation nodes, cannot automatically provide feedback and optimize for re-output, and cannot operate in a closed-loop manner. 3. The linear design has a rigid design logic and cannot dynamically plan workflow paths in response to different input information requirements.
[0055] 4. In the final video merging process of the image-to-video stage, there is a lack of effective merging logic. It is usually a sequential merging, which is not intelligent enough and results in poor generation quality.
[0056] The LangGraph framework provides an easy-to-implement, maintain, and optimize text-based video framework. It constructs a multi-node evaluation design to ensure generation consistency and can perform retry optimization based on evaluation feedback, further improving generation quality. Finally, based on the evaluation results, the model dynamically plans the next execution node, forming a tree-like recall and retry mechanism.
[0057] Example 2 This embodiment proposes a text-based video generation method, which includes the following steps: Assuming the input theme text is "A fat orange cat is lying down eating fish", then the user_params are passed in a comic style with a warm color tone.
[0058] (a) Initialize anchor points Anchor prompts are generated using LLM based on the input information. The prompt content is as follows: Markdown (Subject and Action): A round, fluffy, chubby orange cat lies lazily on a comfortable wooden floor, stretching its body. It is happily devouring grilled fish from a blue ceramic plate. The cat's expression is blissful, its eyes half-closed as it enjoys its meal, with a few crumbs still clinging to its whiskers.
[0059] Environment: A sun-drenched corner of the living room, soft golden sunlight streaming through the window, casting dappled shadows of leaves on the floor. Art Style: High-quality digital comic style, cel-shaded graphics, crisp black outlines, expressive character designs, and anime aesthetics. Color Scheme and Atmosphere: A predominantly warm color palette, featuring amber, golden yellow, and soft orange hues, with cinematic lighting, creating a cozy and soothing atmosphere. Technical Specifications: Masterpiece, highest quality, sharp focus, 4K resolution, high-detail textures.
[0060] Then, based on this prompt, a corresponding anchor image is generated.
[0061] (II) Storyboard Generation Extract all the main information of the anchor point prompt words to obtain dictionary data: { "main_character": "A round, fluffy, chubby orange cat", "character_details": "A blissful expression, eyes squinting happily, whiskers covered in crumbs", "action": "Lying on the floor, lazily stretching its body, devouring its food", "objects": [ "Grilled fish", "Blue ceramic plate" ], "environment": "A sunny corner of the living room, comfortable wooden floors, windows, and the shadows of leaves", "art_style": "High-quality digital comic, cel-shaded, crisp black outlines, anime aesthetics", "color_theme": "Dominant warm color palette (amber, golden yellow, soft orange)", "atmosphere": "Warm, healing, with cinematic lighting featuring soft sunlight", "technical_specs": "Masterpiece, highest quality, 4K resolution, sharp focus"} This data is transformed into a list of storyboard scripts using an LLM model. The list contains four structured storyboards. The first storyboard example is shown below: { "content_outline": "Close-up shot showing the chubby orange cat chewing on grilled fish, emphasizing the pleasure of eating." "character_action_expression": "The orange cat's cheeks are puffed out as it chews vigorously, its whiskers trembling rapidly with its mouth movements, and its eyes are closed in an inverted U-shape (a happy expression)." "camera_movement": "Extreme close-up, with slight camera shake to simulate the force of eating." "background": "A blurred wooden floor background, with the edge of a blue ceramic plate faintly visible." "visual_style": "Cartoon style, high-contrast cel coloring, emphasizing the exaggerated lines of happiness on the face, warm yellow tones." "duration": "2s"} (3) Video generation This assumes that sora2 is used to generate the storyboard video. Since sora2 automatically generates audio content, the TTS module is skipped. Based on the storyboard script information obtained above, the storyboard images are first generated through the VLM module, and then the storyboard images are used to generate short storyboard videos of corresponding durations through the sora2 interface.
[0062] (4) Video post-processing and compositing The storyboard list is passed into the LLM module. Based on the content_outline model in the storyboard information, the storyboards are sorted. Based on the sorting result, the video content logic is assumed to be "(close-up) cat eating fish", "(medium shot) cat's reaction after swallowing", "(overhead panoramic view) showing the cat's body shape and surrounding environment", and "(close-up) cat preparing to sleep". Transitions are generated based on this order. The last frame and the first frame of two adjacent storyboards are sliced and the images are embedded using an image encoding model. Cosine similarity is calculated to determine which transition function to use (VLM generates transitions based on the previous and next frames, or directly introduces fade-in / fade-out and overlay processing). Then, the different storyboard videos are merged into a complete video and saved locally.
[0063] Example 3 See Figure 8 This embodiment proposes a text-based video generation system, including: The first generation module is used to obtain user input parameters and generate anchor prompt words and anchor role diagrams based on the user input parameters. The first evaluation module is used to evaluate the anchor point role diagram until the anchor point role diagram passes the evaluation; The information extraction module is used to extract information from the anchor point prompts after the anchor point character diagram has passed the evaluation, and to obtain the storyboard script. The second evaluation module is used to evaluate the storyboard until the storyboard passes the evaluation. The second generation module is used to generate audio and storyboard video based on the storyboard after the storyboard evaluation is passed. The third evaluation module is used to evaluate the storyboard video until the storyboard video passes the evaluation; The third generation module is used to post-process the storyboard video to obtain a post-processed video, and then combine the post-processed video with the audio to generate the target video.
[0064] First, anchor point prompts and anchor point character diagrams are generated based on user input parameters. Then, the anchor point character diagrams are evaluated to ensure accurate character portrayals, laying a solid foundation for the video. Next, anchor point prompt information is extracted to obtain a storyboard, which is then evaluated to ensure clear script logic and plot coherence. Based on the evaluated script, audio and storyboard video are generated, and the storyboard video is evaluated again to improve image quality and motion smoothness. Finally, the target video is synthesized through post-processing and audio, thereby improving the quality of the generated text-based video. This invention also establishes a closed-loop control system by setting evaluation nodes in the three key stages of anchor point character diagrams, storyboard scripts, and storyboard video, effectively opening up the closed-loop link of text-based video generation and ensuring the controllability and traceability of the generation process.
[0065] Example 4 See Figure 7 This embodiment also proposes a computer device, see [link to relevant documentation]. Figure 9 It includes: a processor 91, a memory 92, a communication interface 93 and a communication bus 94, wherein the processor 91, the memory 92 and the communication interface 93 communicate with each other through the communication bus 94; The processor 91, memory 92, and communication interface 93 communicate with each other via a communication bus 94. The communication interface 93 is used for network communication with other devices, such as clients or other servers. The processor 91 executes executable instructions 95, specifically performing the operations of the text-based video generation method described above. Specifically, the executable instructions 95 may include program code. The processor 91 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention. The computer device includes one or more processors, which may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.
[0066] Memory 92 is used to store executable instructions 95. Memory 92 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.
[0067] Executable instruction 95 can be invoked by processor 91 to cause the computer device to perform the following operations: S1. Obtain user input parameters, and generate anchor prompt words and anchor role diagrams based on the user input parameters; S2. Evaluate the anchor point role diagram until the anchor point role diagram passes the evaluation, then proceed to S3; S3. Extract information from the anchor point prompts to obtain the storyboard; S4. Evaluate the storyboard until the storyboard passes the evaluation, then proceed to S5; S5. Based on the storyboard script, generate audio and storyboard video respectively; S6. Evaluate the storyboard video until the storyboard video evaluation passes, then proceed to S7; S7. Post-process the storyboard video to obtain a post-processed video, and combine the post-processed video with the audio to generate the target video.
[0068] In this embodiment, anchor prompts and anchor character diagrams are first generated based on user input parameters. The anchor character diagrams are then evaluated to ensure accurate character portrayals, laying a solid foundation for the video. Next, anchor prompt information is extracted to obtain a storyboard, which is then evaluated to ensure clear script logic and coherent plot. Based on the evaluated script, audio and storyboard video are generated, and the storyboard video is evaluated again to improve image quality and motion smoothness. Finally, the target video is synthesized through post-processing and audio, thereby improving the quality of the generated text-based video. Furthermore, this invention establishes a closed-loop control system by setting evaluation nodes in the three key stages of anchor character diagrams, storyboard scripts, and storyboard video, effectively connecting the closed-loop chain of text-based video generation and ensuring the controllability and traceability of the generation process.
[0069] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.
Claims
1. A method for generating text-based videos, characterized in that, Includes the following steps: S1. Obtain user input parameters, and generate anchor prompt words and anchor role diagrams based on the user input parameters; S2. Evaluate the anchor point role diagram until the anchor point role diagram passes the evaluation, then proceed to S3; S3. Extract information from the anchor point prompts to obtain the storyboard; S4. Evaluate the storyboard until the storyboard passes the evaluation, then proceed to S5; S5. Based on the storyboard script, generate audio and storyboard video respectively; S6. Evaluate the storyboard video until the storyboard video evaluation passes, then proceed to S7; S7. Post-process the storyboard video to obtain a post-processed video, and combine the post-processed video with the audio to generate the target video.
2. The text-based video generation method according to claim 1, characterized in that, The step of generating anchor prompts and anchor role diagrams based on the user input parameters includes: S11. Determine whether the user input parameters contain a custom image. If yes, proceed to S12; otherwise, proceed to S13. S12. Input the user input parameters into the preset large language model, and generate anchor prompt words from the output image of the large language model, and skip generating the anchor role diagram; S13. Input the user input parameters into a preset large language model, and the large language model outputs an image to generate anchor point prompts. Input the image-generated anchor point prompts into a preset visual language model, and the visual language model outputs an anchor point role diagram.
3. The text-based video generation method according to claim 2, characterized in that, The process of evaluating the anchor point role diagram until the anchor point role diagram passes the evaluation includes: The user input parameters, the anchor prompt words, and the anchor role diagram are input into the visual language model. The visual language model analyzes whether the core elements in the anchor role diagram match the anchor prompt words. If they match, the anchor role diagram passes the evaluation; if not, the anchor role diagram fails the evaluation. The reason for the failure is input into the large language model, which then regenerates the anchor prompt words.
4. The text-based video generation method according to claim 3, characterized in that, The step of extracting information from the anchor cue words to obtain the storyboard includes: inputting the anchor cue words into the large language model, extracting the main information of the anchor cue words from the large language model, and generating a structured storyboard based on the main information.
5. The text-based video generation method according to claim 4, characterized in that, The process of evaluating the storyboard until it passes evaluation includes: S41. Encapsulate the storyboard script according to a preset format to obtain a storyboard preset format script; S42. Input the storyboard preset format script into the large language model. Use the large language model to analyze whether there is a conflict between the content outline and the visual style of the storyboard preset format script. If so, the storyboard script fails the evaluation. Output the storyboard index and the reason for the conflict. Input the storyboard index, the reason for the conflict, and the anchor point prompt into the large language model. The large language model regenerates the storyboard script. Continue until the regenerated storyboard script fails the evaluation several times. Then, directly select the storyboard script to participate in the last evaluation and execute S5. If not, the storyboard script passes the evaluation and execute S5.
6. The text-based video generation method according to claim 2, characterized in that, The process of generating audio and storyboard video based on the storyboard script includes: S51. Input the storyboard into a preset text-to-speech model, and have the text-to-speech model output the audio. S52. Fill the storyboard script, the user input parameters and the anchor character diagram into the storyboard image generation prompt, input the storyboard image generation prompt into the large language model, and output the storyboard image from the large language model; S53. Input the storyboard images and video generation prompts into a preset video generation model, and output the storyboard video from the video generation model.
7. The text-based video generation method according to claim 6, characterized in that, The evaluation of the storyboard video until the storyboard video evaluation passes includes: S61. Input the storyboard video and the video-generated prompts into the large language model. The large language model performs a binary evaluation based on three dimensions: subject consistency, story integrity, and video performance, and outputs a binary evaluation result, which is 0 or 1. S62. Sum the results of the two-factor evaluation to obtain the total evaluation score; S63. Determine whether the total evaluation score is equal to 3. If yes, the storyboard video evaluation is passed; otherwise, the storyboard video evaluation is not passed.
8. The text-based video generation method according to claim 7, characterized in that, The post-processing of the storyboard video to obtain a post-processed video includes: S71. Use the large language model to sort the storyboard videos to obtain the sorted storyboard videos; S72. Extract the last frame of the previous storyboard video and the first frame of the next storyboard video from the storyboard video sequence; S73. Using a preset image feature extraction model, extract the first embedding vector of the last frame of the previous storyboard video and the second embedding vector of the first frame of the next storyboard video, and calculate the cosine similarity between the first embedding vector and the second embedding vector. S74. Compare the cosine similarity with the transition threshold to obtain the comparison result, and select different transitions to generate a transition video based on the comparison result; S75. Merge the storyboard video and the transition video to obtain a merged video, and then splice the merged video with the audio to obtain the target video.
9. A text-based video generation system, characterized in that, include: The first generation module is used to obtain user input parameters and generate anchor prompt words and anchor role diagrams based on the user input parameters. The first evaluation module is used to evaluate the anchor point role diagram until the anchor point role diagram passes the evaluation; The information extraction module is used to extract information from the anchor point prompts after the anchor point character diagram has passed the evaluation, and to obtain the storyboard script. The second evaluation module is used to evaluate the storyboard until the storyboard passes the evaluation. The second generation module is used to generate audio and storyboard video based on the storyboard after the storyboard evaluation is passed. The third evaluation module is used to evaluate the storyboard video until the storyboard video passes the evaluation; The third generation module is used to post-process the storyboard video to obtain a post-processed video, and then combine the post-processed video with the audio to generate the target video.
10. A computer device, characterized in that, include: The processor, memory, communication interface, and communication bus are provided, wherein the processor, memory, and communication interface communicate with each other via the communication bus. The memory is used to store at least one executable instruction that causes the processor to perform the operation of the text-based video generation method as described in any one of claims 1-8.