A video generation method and related apparatus

Abstract

The application discloses a video generation method and related device, by dividing the video generation process into N generation stages after obtaining the video description text used for describing the target video, the target video is generated. By displaying the generated content and providing the to-be-evaluated question when the interactive condition of the i-th generation stage is reached, the user can interact with the video generation process, by displaying the generated content of the i-th generation stage, the user can more timely find the deviation between the generated content and the demand, and adjust or confirm through the feedback information, so that the video generation process has stronger controllability. In response to the feedback information being the acceptance of the generated content, the video generation is continued based on the generated content, and in response to the feedback information being the content adjustment instruction, the video generation of the i-th generation stage is re-performed based on the content adjustment instruction. Since only the current generation stage needs to be re-performed for video generation, the complete video is avoided from being regenerated, and the computational overhead is reduced.

Description

Technical Field

[0001] This application relates to the field of media processing, and in particular to a video generation method and related apparatus. Background Technology

[0002] Currently, deep learning-based video generation technology is maturing, allowing users to generate videos by inputting prompts into the video generation model.

[0003] In related technologies, a one-time generation method is usually adopted. For example, after receiving the prompt words, the complete video is obtained directly. If the user is not satisfied with the video, the prompt words need to be modified.

[0004] However, for one-time video generation methods, improving the accuracy of the generated video usually requires users to increase the precision of the prompts. For example, by adding descriptive details or limiting generation conditions, users can guide the video generation model to generate a video that meets expectations. Users need to provide accurate prompts all at once before generation, resulting in a high barrier to entry. Moreover, even if the prompts are relatively accurate, the video generation model may still deviate during execution, making it difficult to ensure that the generated video meets user expectations, resulting in low accuracy. Summary of the Invention

[0005] To address the aforementioned technical problems, this application provides a video generation method and related apparatus to lower the barrier to entry for users and improve the accuracy of video generation.

[0006] The embodiments of this application disclose the following technical solutions: On one hand, embodiments of this application provide a video generation method, the method comprising: Obtain the video description text used to describe the target video; Based on the video description text, video generation is performed sequentially in N stages for the target video, where N>1; The target video is obtained through video generation in the Nth generation stage; In the video generation of the N generation stages: When generating video in the i-th generation stage, in response to the achievement of the interaction condition, the generated content of the i-th generation stage at the time the interaction condition is met and the evaluation questions for the generated content are displayed, 1≤i <N; In response to the feedback information of the problem to be evaluated, the generated content is accepted and video generation in the i-th generation stage is continued based on the generated content, or video generation in the (i+1)-th generation stage is performed based on the generated content; In response to the feedback information of the problem to be evaluated, a content adjustment instruction is given for the generated content, and the video generation in the i-th generation stage is performed again based on the content adjustment instruction.

[0007] On the other hand, embodiments of this application provide a video generation apparatus, the apparatus including an acquisition unit and a generation unit; The acquisition unit is used to acquire video description text used to describe the target video; The generation unit is configured to sequentially generate video for the target video in N generation stages based on the video description text, where N>1; The generation unit is further configured to obtain the target video through video generation in the Nth generation stage; In the video generation process involving the N generation stages, the generation unit includes a display unit, a continued generation unit, and a regeneration unit: The display unit is configured to, when video generation is performed in the i-th generation stage, display the generated content of the i-th generation stage at the time the interaction condition is met, and the evaluation question for the generated content, 1≤i, in response to the achievement of the interaction condition. <N; The continuing generation unit is configured to respond to the feedback information of the problem to be evaluated as accepting the generated content, and continue video generation in the i-th generation stage based on the generated content, or perform video generation in the (i+1)-th generation stage based on the generated content; The regeneration unit is configured to respond to the feedback information of the problem to be evaluated as a content adjustment instruction for the generated content, and regenerate the video in the i-th generation stage based on the content adjustment instruction.

[0008] On the other hand, embodiments of this application provide a computer device, the computer device including a processor and a memory: The memory is used to store computer programs and to transfer the computer programs to the processor; The processor is configured to execute the methods described above according to instructions in the computer program.

[0009] On the other hand, embodiments of this application provide a computer-readable storage medium for storing a computer program for performing the methods described above.

[0010] On the other hand, embodiments of this application provide a computer program product including a computer program, which, when run on a computer device, causes the computer device to perform the methods described above.

[0011] As can be seen from the above technical solution, this application divides the video generation process into N generation stages after obtaining the video description text used to describe the target video, and introduces a mechanism for users to adjust the content during the generation process, thereby achieving progressive video generation. Taking the i-th generation stage as an example, by displaying the generated content and providing evaluation questions when the interaction conditions are met in the i-th generation stage, users can interact with the video generation process. Compared with the relatively closed black-box generation mode in related technologies, by displaying the generated content of the i-th generation stage, users can more promptly discover the deviation between the generated content and their needs from a visual perspective, and make adjustments or confirmations based on the evaluation questions, making the video generation process more controllable for users. After the user provides feedback based on the information to be evaluated, in response to the feedback indicating acceptance of the generated content, video generation continues in the current stage or proceeds to the next stage, allowing the confirmed generated content to be used in subsequent video generation processes. In response to the feedback indicating content adjustment, video generation in the i-th stage is restarted based on the content adjustment instruction. Since only the current generation stage needs to be regenerated, the entire video is avoided, reducing computational overhead. Thus, by dividing the generation process into multiple stages and introducing a user feedback mechanism, deviations in the generated content can be corrected in a timely manner. This not only reduces the probability of these deviations propagating in subsequent stages and improves the consistency between the target video and user expectations, thereby increasing video accuracy, but also allows users to visually clarify their needs for the video through progressive confirmation and adjustment. This eliminates the need for users to provide precise video descriptions before video generation, lowering the barrier to entry for users. Furthermore, because users can provide feedback to continuously confirm and adjust the video generation process, the probability of the target video generated in one go meeting the user's needs is higher. In other words, a more accurate video can be obtained without modifying the prompts multiple times and generating the video repeatedly. This reduces the number of times the video is generated, reduces the waste of computing resources, and improves resource utilization. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 A schematic diagram of a computer system for a video generation method provided in an embodiment of this application; Figure 2This is a schematic diagram illustrating an application scenario of a video generation method provided in an embodiment of this application; Figure 3 This is an interactive schematic diagram of a terminal device provided in an embodiment of this application; Figure 4 A flowchart illustrating a video generation method provided in an embodiment of this application; Figure 5 A schematic diagram of an interactive page provided in an embodiment of this application; Figure 6 This application provides an example of an interactive page in a multi-select scenario. Figure 7 An interactive page diagram for adjusting content annotations is provided as an embodiment of this application; Figure 8 A schematic diagram of a system architecture provided for an embodiment of this application; Figure 9 A flowchart based on interactive checkpoints is provided for an embodiment of this application; Figure 10 This application provides a schematic diagram of a complete video generation process. Figure 11 This is one of the timing flow diagrams of interactive checkpoints provided in the embodiments of this application; Figure 12 The second timing flowchart of the interactive checkpoint provided in the embodiments of this application; Figure 13 The third schematic diagram of the timing flow of interactive checkpoints provided in the embodiments of this application; Figure 14 Fourth of the timing flow diagrams for interactive checkpoints provided in the embodiments of this application; Figure 15 Fifth of the timing flowcharts for interactive checkpoints provided in the embodiments of this application; Figure 16 Sixth schematic diagram of the timing flow of interactive checkpoints provided in the embodiments of this application; Figure 17 Seventh of the timing flow diagrams for interactive checkpoints provided in the embodiments of this application; Figure 18 Eighth of the timing flow diagrams for interactive checkpoints provided in the embodiments of this application; Figure 19 This is a schematic diagram of the structure of a video generation device provided in an embodiment of this application; Figure 20 A structural diagram of a terminal device provided in an embodiment of this application; Figure 21 This is a structural diagram of a server provided in an embodiment of this application. Detailed Implementation

[0014] The embodiments of this application will now be described with reference to the accompanying drawings.

[0015] In related technologies, videos are typically generated in a single step. The following are two examples illustrating different video generation methods: (a) One-time video generation system This is the model used by the vast majority of video generation products on the market. After the user inputs a text prompt or reference image, the video generation system completes all generation calculations in one go and outputs a complete video. If the user is not satisfied, they can only modify the prompt and regenerate from scratch.

[0016] In this video generation method, the only control a user has is to write better prompts or use reference images to constrain the generation style. The generation process itself is completely invisible and uninterrupted to the user.

[0017] (II) Technical solutions to enhance control at the input end

[0018] In recent years, several technologies have emerged that add control dimensions at the input end. For example, ControlNet technology allows users to guide generation by providing additional conditions such as edge maps, depth maps, and human pose maps; IP-Adapter allows users to control the generation style and character appearance using reference images; and InteractiveVideo allows users to control the content and direction of movement of regions through graffiti annotations and motion trajectory drawing.

[0019] However, the common feature of the above technical solutions is that all user control actions occur before video generation begins, which is a pre-set conditional control. Once generation begins, the user can no longer intervene.

[0020] Based on this, this application proposes a video generation method and related apparatus. By dividing the video generation process into multiple generation stages and displaying generated content and evaluation questions based on interactive conditions during the generation process, users can participate in the intermediate generation process of the video. This breaks the black-box, one-time generation method in related technologies, allowing users to promptly discover deviations between the generated content and their needs and make confirmations or adjustments. This improves the controllability of the video generation process, enhances the consistency between the target video and user expectations, and reduces users' reliance on the accuracy of prompts by progressively clarifying requirements, thereby lowering the barrier to entry.

[0021] To facilitate understanding of the video generation method provided in the embodiments of this application, the computer system of the video generation method will be described first below.

[0022] See Figure 1This figure is a schematic diagram of a computer system for a video generation method provided in an embodiment of this application. The computer system 100 includes multiple devices, such as multiple terminal devices 110 and multiple servers 120, etc. The terminal devices 110 and the servers 120 can communicate with each other through a communication network.

[0023] The communication network uses standard communication technologies and / or protocols, typically the Internet, but can also be any network, including but not limited to Bluetooth, local area network (LAN), metropolitan area network (MAN), wide area network (WAN), mobile, private network, or any combination of virtual private network. In some embodiments, custom or dedicated data communication technologies may be used to replace or supplement the aforementioned data communication technologies.

[0024] The video generation method provided in this application can be implemented using a computer device, which can be a terminal device or a server. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. Terminal devices include, but are not limited to, mobile phones, computers, smart voice interaction devices, smart home appliances, vehicle terminals, aircraft, and extended reality (XR) devices. The terminal device and the server can be directly or indirectly connected via wired or wireless communication, and this application does not impose any limitations on this connection.

[0025] It is understood that in the specific implementation of this application, data such as user information, video description text, and feedback information are involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0026] To facilitate understanding of the video generation method provided in this application embodiment, the following example uses a server as the execution subject of the video generation method to illustrate its application scenarios.

[0027] See Figure 2This figure is a schematic diagram of an application scenario for a video generation method proposed in an embodiment of this application. The application scenario includes a terminal device 210 and a server 220. The terminal device 210 may be one of the aforementioned multiple terminal devices 110, and the terminal device 210 is used to interact with the user during the video generation process. The server 220 may be one of the aforementioned multiple servers 120, and the server 220 has deployed a video generation model, which is used to generate video in multiple stages.

[0028] See Figure 3 This figure is a schematic diagram of the interaction of a terminal device proposed in an embodiment of this application. The following example illustrates video generation through two stages. Figure 3 For a detailed explanation of this application scenario, please refer to A1-A5: In A1, terminal device 210 acquires video description text used to describe the target video. For example... Figure 3 As shown, users can enter the video description text "a cat jumps from the ground onto a table" in the input box below the text description.

[0029] In A2, the terminal device 210 sends the video description text to the server 220 via the communication network, and can also input the video description text into the deployed video generation model.

[0030] In A3, when the video is generated in the first generation stage, in response to the achievement of the interaction conditions, the terminal device 210 displays the generated content of the first generation stage when the interaction conditions are achieved and the questions to be evaluated for the generated content.

[0031] Specifically, after the video description text is input into the video generation model, server 220 performs the first generation stage of video generation. Taking the completion of the first generation stage as an example, the generated content of the first generation stage can be generated at the end of the first generation stage, including the character settings and scene settings of the target video. The appearance of the cat is determined by the character settings, and the style of the table is determined by the scene settings. It should be noted that the question to be evaluated can also be generated by the question generation model.

[0032] In addition, evaluation questions can be generated for character or scene settings. Server 220 can send the generated content at the end of the first generation stage and the evaluation questions for the generated content to terminal device 210 via a communication network, so that terminal device 210 can display them.

[0033] like Figure 3As shown, the issue to be evaluated is "Does the appearance of this cat match the description?", and options A "Matches, Confirmed" and B "Inconsistent, Needs Modification" are provided for the user to choose from.

[0034] In A4, terminal device 210 can obtain feedback information about the problem to be evaluated, such as... Figure 3 As shown, the user can select option B. After selecting option B, the user can submit modification suggestions through terminal device 210, such as... Figure 3 The image shows "The cat is getting a little fatter, and its face is rounder."

[0035] After receiving the feedback information, the terminal device 210 can send it to the server 220 via the communication network.

[0036] In A5, in response to the feedback information of the problem to be evaluated, which is a content adjustment instruction for the generated content, the terminal device 210 regenerates the video in the second generation stage based on the content adjustment instruction to obtain the target video.

[0037] Server 220 can input video description text and feedback information into the video generation model to perform the second generation stage of video generation and obtain the target video.

[0038] Specifically, terminal device 210 can send the modification suggestions to server 220 via the communication network to obtain the modified character settings (this step is not shown in the figure), and display the evaluation question "Does the appearance setting of this cat match the description?" for the user to confirm again. Taking the user selecting option A "Matching, Confirm Pass" as an example, the subsequent generation process can continue based on the scene settings and the modified character settings to obtain the target video.

[0039] Terminal device 210 sends the target video to itself via a communication network. For example... Figure 3 As shown, terminal device 210 can display the target video.

[0040] Therefore, by displaying the generated content and obtaining feedback information in the first generation stage, users can confirm or adjust character and scene settings during video generation. This prevents content that does not meet expectations from spreading in subsequent generation stages, improving user control over the video generation process and making the target video generated in one go more in line with user needs. Furthermore, since only the character settings need to be regenerated after receiving instructions to adjust the character settings, instead of regenerating the entire video, computational resource consumption is reduced.

[0041] The video generation method provided in this application can also be applied to various application scenarios, which are illustrated below through five application scenarios.

[0042] (1) Preview production for film and animation. In the early stages of a film or television project, the director and production team need to quickly generate concept preview videos (pre-visualization) to verify the creative direction. The video generation method provided in this application can divide the preview video generation process into multiple stages. The director can gradually confirm each stage, such as character setting, scene style, and motion choreography, to ensure that the final preview video accurately conveys the creative intent and avoids repeated revisions.

[0043] (2) Short video and social media content creation. Creators need to efficiently generate attractive video content. The video method provided in this application enables creators to adjust elements such as character actions, screen rhythm, and camera movement in real time by customizing interactive conditions during the generation process, completing the entire process from creative conception to final output in a few minutes, without having to generate videos multiple times by adjusting prompts, which is highly efficient.

[0044] (3) Generation of e-commerce product display videos. E-commerce operators need to create display videos for a large number of products. The video generation method provided in this application enables operators to confirm the correct rendering of the product appearance before generating subsequent scenes and actions during the product appearance generation stage, thus avoiding the entire video being scrapped due to inaccurate product rendering.

[0045] (4) Production of Educational and Training Content. Educators need to produce instructional videos that demonstrate specific knowledge points. The video generation method provided in this application enables teachers to verify the accuracy of actions and procedures during the key knowledge demonstration stages of the instructional video production process, ensuring the professionalism and correctness of the teaching content.

[0046] (5) Production of advertising and marketing videos. Advertising creative teams need to produce multiple versions of advertising videos within a limited budget. The video generation method provided in this application supports advertising creative teams in creating multiple branches at key creative nodes (such as brand element presentation, product close-ups, actor actions, etc.), thereby producing multiple versions of videos during the video generation process and making precise control, without having to repeatedly generate multiple complete videos, improving the first-time success rate and reducing costs.

[0047] It should be noted that the above application scenarios are merely examples, and the video generation method provided in this embodiment can also be applied to other scenarios, which are not limited here.

[0048] The following describes in detail a video generation method provided in this application through method embodiments.

[0049] See Figure 4This figure is a schematic flowchart of a video generation method provided in an embodiment of this application. For ease of description, the following embodiment uses a terminal device as the executing entity of the video generation method. Figure 4 As shown, the video generation method includes S401-S403: S401: Obtain the video description text used to describe the target video.

[0050] The target video is the complete video ultimately generated by the video generation model. The video description text is a textual description of the video content. The video description text can be used to describe objects, scenes, actions, plot, style effects, or other content that can describe the video. For example, the video description text could be "An orange cat jumps from the table to the ground and then leisurely walks to the balcony. Indoor scene, afternoon sunlight, warm atmosphere."

[0051] In this embodiment, the video description text is used to describe the target video. That is, the user can express their generation needs for the target video through the video description text, enabling the video generation model to understand the user's intent and generate the video accordingly. The video generation model is a model used to generate videos.

[0052] The embodiments of this application do not specifically limit the technical implementation of the video generation model. For example, the diffusion model can be a video generation model based on a diffusion transformer (DiT), a video generation model based on a U-shaped network (UNet) diffusion model, a video generation model based on an autoregressive transformer (AR transformer), a video generation model based on a generative adversarial network (GAN), etc.

[0053] In one possible implementation, the video generation model can satisfy the following interface conditions: (a) Supports pausing and resuming the generation process (or supports segmented calls); (b) Supports receiving external condition constraints as generation guidance; (c) It can output the stage results of each generation stage.

[0054] Currently, most video generation models meet these conditions, enabling the video generation method proposed in this application to have good compatibility.

[0055] As one implementation method, in addition to obtaining the video description text, reference images used to describe the target video can also be obtained. The reference images are images used to reflect the video content. For example, the reference images can be schematic images of character appearance, schematic images of scene layout, schematic images of screen style, etc., so that the video generation model can generate video based on the video description text and reference images.

[0056] S402: Based on the video description text, perform video generation in N generation stages for the target video in sequence.

[0057] As mentioned above, video generation is the process of inputting video description text into a video generation model so that the video generation model can generate the corresponding video.

[0058] In this embodiment of the application, the video generation is divided into N generation stages and generated sequentially to achieve progressive video generation, where N>1.

[0059] The various generation stages not only have a sequential execution relationship in time, but also a clear execution dependency relationship, that is, the input of the subsequent generation stage depends on the stage results of the preceding generation stage.

[0060] Specifically, during execution, the result of the i-th generation stage is used as the input of the (i+1)-th generation stage, forming a chain-like transmission relationship between the generation stages. This ensures that the result of each stage is in a verifiable and adjustable state before entering the next stage, allowing the user to adjust it to reduce deviations from continuing into subsequent generation stages.

[0061] For example, the subsequent video clip generation stage needs to generate videos based on the character and scene settings determined in the previous stage.

[0062] The generation stage is a segmentation of the video generation process, with each stage having its own corresponding results. These results represent the generated content of the target video. Unlike the complete target video, the results for the first N-1 generation stages visually break down the target video generation process into multiple nodes from specific dimensions.

[0063] Taking the video description text "An orange cat jumps from the table to the ground and then leisurely walks to the balcony. Indoor scene, afternoon sunlight, warm atmosphere" as an example, in the character appearance generation stage, an image of the orange cat can be generated, and in the motion trajectory generation stage, the motion trajectory of the orange cat jumping from the table to the ground and walking to the balcony can be generated.

[0064] The embodiments of the present application do not specifically limit the deployment location of the video generation model, and the video generation model can be deployed on a server or a terminal device. If the video generation model is deployed on the server, the terminal device can transmit the obtained video description text to the server, and through the video generation model of the server, perform video generation in N generation stages for the target video in sequence, and then transmit the stage results of the N generation stages to the terminal device in sequence for display by the terminal device. If the video generation model is deployed on the terminal device, the terminal device can directly input the obtained video description text into the video generation model, perform video generation in N generation stages for the target video in sequence, and display the results of each stage in sequence.

[0065] S403: Obtain the target video through video generation in the Nth generation stage.

[0066] The Nth generation stage is the last generation stage, and the stage result of the Nth generation stage is the target video. After completing the video generation in the Nth generation stage, the target video can be obtained.

[0067] Taking the ith generation stage in the first N - 1 generation stages as an example below, the video generation process of the generation stage will be described, where 1 ≤ i < N. Specifically, refer to S4021 - S4023: S4021: When performing video generation in the ith generation stage, in response to reaching the interaction condition, display the generation content at the time of reaching the interaction condition in the ith generation stage and the evaluation questions for the generation content.

[0068] The interaction condition is a condition used to determine the timing of triggering user participation in the interaction during the video generation process, that is, it can be determined whether the current generation content needs to be confirmed or adjusted by the user through the interaction condition. By introducing the interaction condition, the video generation process changes from continuous execution to a process that can pause at key nodes and guide user participation, so that the user can intervene in the stage - by - stage generation content during the generation process, improving the controllability of the video generation process, and thus improving the consistency between the target video and the user's expectations.

[0069] The interaction condition can be specified by the user, such as "pause generation when generating the appearance of a kitten", and the interaction condition can also be pre - set, such as pausing generation when the semantic deviation between the generation content of the video generation and the video description text is large. The embodiments of the present application do not limit this.

[0070] In response to the achievement of the interaction condition, the video generation process of the i-th generation stage is paused, and the generated content of the i-th generation stage at the time of the interaction condition is displayed, along with the evaluation questions for the generated content. The generated content represents the stage-wise generation result of the target video, and the evaluation questions are questions constructed based on the generated content to guide the user in accepting or adjusting the content. For example, does the cat's appearance match the description? Does the scene style meet expectations? Does the movement speed need adjustment? Please select the more expected solution A or solution B.

[0071] For example, the generated content at the i-th generation stage when the interaction condition is met can be obtained as follows: First, the intermediate generation state of the video generation model at the time the interaction condition is met is obtained. The intermediate generation state is decoded into a visualization result, and then the generated content corresponding to the i-th generation stage is extracted based on the visualization result. The intermediate generation state can be the latent variable representation of the diffusion model during the denoising process, the latent space features corresponding to the current denoising step, the latent representation of intermediate frames, etc.

[0072] This transforms the previously continuous video generation process into a pauseable and interactive one, allowing users to directly observe the generated content at a certain stage. This breaks the black-box approach to video generation, making the generated content visible and evaluable to users, and providing a basis for subsequent confirmation or adjustments.

[0073] S4022: In response to the feedback information of the problem to be evaluated, the generated content is accepted, and video generation in the i-th generation stage is continued based on the generated content, or video generation in the (i+1)-th generation stage is performed based on the generated content.

[0074] Feedback information describes the user's response to the issue to be evaluated. Feedback includes acceptance of video content and content adjustment instructions. Acceptance of video content indicates that the user accepts the generated content upon meeting the interaction conditions; this video content can be used to continue subsequent video generation. Content adjustment instructions represent user instructions to modify the generated content. Examples include, "Make the cat a little fatter, with a rounder face," "Change the scene to nighttime," "Reduce the movement speed," and "Regenerate a specific area."

[0075] If the feedback message for the question to be evaluated is "accept the generated content," it indicates that the generated content is largely consistent with the user's expectations, and video generation can continue based on the generated content. For example, if the interaction condition is met while the user is in the i-th generation stage, video generation in the i-th generation stage can continue based on the generated content; if the interaction condition is met at the end of the i-th generation stage, video generation in the (i+1)-th generation stage can proceed based on the generated content.

[0076] Therefore, by using the confirmed generated content in the subsequent video generation process, the subsequent generation process is based on generated content that meets the user's expectations, making the subsequent video process more consistent with the user's expectations.

[0077] S4023: In response to the feedback information of the problem to be evaluated, which is a content adjustment instruction for the generated content, the video generation of the i-th generation stage is restarted based on the content adjustment instruction.

[0078] If the feedback information for the problem to be evaluated is a content adjustment instruction for the generated content, indicating that the generated content deviates from the user's expectations, then the video generation in the i-th generation stage can be restarted based on the content adjustment instruction.

[0079] For example, during the character design generation phase, if a user thinks the cat is too thin and suggests that the cat be a little fatter, the cat's appearance can be regenerated based on this suggestion to create a more proportionate cat character design.

[0080] In other words, in this embodiment, instead of simply requiring the user to re-enter the video description text and re-initiate a complete video generation task, the video generation in the i-th generation stage is directly restarted based on the content adjustment instructions. This limits the user's modifications to the current generation stage. Compared to related technologies where users can only trigger the regeneration of a complete video by modifying the prompts, this approach not only allows for timely detection and feedback of deviations in the i-th generation stage, improving the user's controllability and visibility over the video generation process, but also enables adjustments to local generated content, enhancing the correlation between the content adjustment instructions and the generated content, and improving the targeting of the adjustments.

[0081] In this embodiment, the video generation process is transformed from a one-time generation to a progressive generation process that proceeds step-by-step according to generation stages. In each generation stage, by displaying the generated content and obtaining feedback information when interactive conditions are met, the user can progressively confirm or adjust the generated content at each stage. This transforms the overall uncontrollable generation process of the target video into a stage-by-stage controllable one, improving the accuracy of the target video by gradually approaching the user's expectations. Compared to the method of generating videos multiple times in related technologies, a single video generation process can generate a more accurate video that meets the user's expectations, reducing the computational process of repeated video generation, lowering computational resource consumption, improving video generation efficiency, and thus improving resource utilization.

[0082] As can be seen from the above technical solution, this application divides the video generation process into N generation stages after obtaining the video description text used to describe the target video, and introduces a mechanism for users to adjust the content during the generation process, thereby achieving progressive video generation. Taking the i-th generation stage as an example, by displaying the generated content and providing evaluation questions when the interaction conditions are met in the i-th generation stage, users can interact with the video generation process. Compared with the relatively closed black-box generation mode in related technologies, by displaying the generated content of the i-th generation stage, users can more promptly discover the deviation between the generated content and their needs from a visual perspective, and make adjustments or confirmations based on the evaluation questions, making the video generation process more controllable for users. After the user provides feedback based on the information to be evaluated, in response to the feedback indicating acceptance of the generated content, video generation continues in the current stage or proceeds to the next stage, allowing the confirmed generated content to be used in subsequent video generation processes. In response to the feedback indicating content adjustment, video generation in the i-th stage is restarted based on the content adjustment instruction. Since only the current generation stage needs to be regenerated, the entire video is avoided, reducing computational overhead. Thus, by dividing the generation process into multiple stages and introducing a user feedback mechanism, deviations in the generated content can be corrected in a timely manner. This not only reduces the probability of these deviations propagating in subsequent stages and improves the consistency between the target video and user expectations, thereby increasing video accuracy, but also allows users to visually clarify their needs for the video through progressive confirmation and adjustment. This eliminates the need for users to provide precise video descriptions before video generation, lowering the barrier to entry for users. Furthermore, because users can provide feedback to continuously confirm and adjust the video generation process, the probability of the target video generated in one go meeting the user's needs is higher. In other words, a more accurate video can be obtained without modifying the prompts multiple times and generating the video repeatedly. This reduces the number of times the video is generated, reduces the waste of computing resources, and improves resource utilization.

[0083] This application provides an interactive page in its embodiments. See also... Figure 5 The figure is a schematic diagram of an interactive page proposed in an embodiment of this application.

[0084] This interactive page includes an input panel, a preview area, and an interactive question and feedback area. These areas are interconnected through the execution process of the generation phase, allowing users to input, observe, and provide feedback on the content generated in the current phase.

[0085] The input panel may include a text description input module, a reference image input module, a style preference setting module, and a checkpoint setting module. The text description input module is used to obtain video description text, such as "a cat jumps from the ground onto a table." The reference image input module is used to obtain reference images. The style preference input module provides multiple selectable style preferences for the user to choose from. The user can use the checkpoint setting module to select the type of strategy to trigger the interaction, such as fixed-stage triggering, uncertainty triggering, semantic change triggering, or custom triggering. Related details can be found in the embodiments corresponding to the following four triggering strategies, which will not be elaborated upon here.

[0086] The preview area can be used to display the generated content of each generation stage (such as stage results or intermediate generated content), such as the role setting and scene setting in the global planning stage, and the keyframe sequence in the keyframe generation stage. Once the user has determined the stage result of a particular generation stage, they can lock that stage's result. When an interaction condition (checkpoint trigger) is met during the video generation process of a particular generation stage, the video generation process for that stage will be paused.

[0087] The interactive question and feedback area is used to display questions to be evaluated and to obtain user feedback. For example, for fixed-stage triggers, a question could be asked, "Does the appearance of this cat match the description?" Or, for uncertain triggers, a question could be asked, "The confidence level of keyframe 2 is low; please select a candidate solution."

[0088] This application does not specifically limit the triggering strategy for interaction conditions. The following examples illustrate four triggering strategies.

[0089] Triggering strategy one: fixed phase triggering.

[0090] In trigger strategy one, the interaction condition is the completion of video generation in the i-th generation stage; that is, the interaction condition is met when the i-th generation stage is completed. For details of trigger strategy one, see B1-B2, where B1 is a specific implementation of S4021 (the step of displaying the generated content and the question to be evaluated), and B2 is a specific implementation of S4022 (the step of continuing video generation). B1: Displays the stage results output by the i-th generation stage and the evaluation questions for the stage results.

[0091] As mentioned above, each generation stage has a corresponding stage result, which represents the generated content of the target video. Upon completion of the video generation in the i-th generation stage, the stage result can be displayed, along with an evaluation question regarding the stage structure. This allows users to view the stage result of the i-th generation stage and evaluate its effectiveness. The evaluation question is a type of question to be evaluated, used to guide users in assessing the effectiveness of the stage result. For example, if the stage result of the second generation stage is the character design for the target video, the corresponding evaluation question could be "Is this character design acceptable?" Users can provide feedback by answering "Yes" or "No, it needs modification."

[0092] B2: In response to the feedback information of the effect evaluation question, the results of the acceptance stage are used to generate the video in the (i+1)th generation stage based on the stage results.

[0093] After the user provides feedback on the effect evaluation, if the feedback message is "accept the stage result," it indicates that the stage result is largely consistent with the user's expectations. Therefore, video generation for the (i+1)th generation stage can proceed based on this stage result. For example, the video generation model can be resumed, or the stage result of the ith generation stage can be input into the video generation process to generate the video for the (i+1)th generation stage.

[0094] The feedback information in response to the effect evaluation issue is a content adjustment instruction for the stage results. If the results of this stage deviate from the user's expectations, the video generation of the i-th generation stage can be restarted based on the content adjustment instruction.

[0095] For example, after the character setting generation stage is completed, if the user provides a content adjustment instruction indicating that the cat's fur is too sparse and suggests that the cat's fur could be denser, then this content adjustment instruction can be input into the video generation model, and the cat's character setting can be regenerated to generate a cat with denser fur.

[0096] Therefore, by triggering interaction upon completion of the i-th generation stage, users can confirm or adjust the results of each stage upon completion. This ensures that subsequent generation stages build upon the expected results, reducing deviations in video generation. Furthermore, since the displayed stage results pertain to the content generated within the same stage, users can focus their judgments on the content dimensions corresponding to that stage, thereby improving the clarity of their understanding of the generated content and the accuracy of their evaluation.

[0097] In one possible implementation, this application proposes a method of generating multiple branches at the end of the generation phase for user selection and interaction, as detailed in B11-B21, where B11 is a specific implementation of B1 and B21 is a specific implementation of B2. B11: Displays the L stage results output by the i-th generation stage and the effect evaluation questions for each of the L stage results.

[0098] Where L>1. After the i-th generation stage is completed, L different stage results are generated, and these stage results and their corresponding effect evaluation questions are displayed for the user to select.

[0099] For example, in the character design phase, three different cat appearances are generated and displayed: an orange cat, a black cat, and a white cat, with the prompt "Please select the appearance that best matches the description." Similarly, in the scene generation phase, three scenes are generated: an indoor desktop scene, an outdoor grassy scene, and a nighttime street scene, each accompanied by a corresponding effect evaluation question to prompt the user to make a selection.

[0100] In one possible implementation, L stage results are generated in response to the initial stage result of the i-th generation stage reaching a confidence condition relative to the content confidence of the video description text.

[0101] The initial stage result is the stage result initially generated in the i-th generation stage. If the confidence level of the initial stage result relative to the content of the video description text reaches the confidence condition, it indicates that the probability of the initial stage result not meeting the user's expectations is relatively high. Therefore, L stage results can be generated based on the initial stage result.

[0102] For example, in the character creation stage, if the character appearance generated in the initial stage deviates semantically from the video description text, such as the color of the generated cat being inconsistent with the video description text, then based on the initial stage result, constraints on color can be introduced so that the video generation model can generate cats of multiple colors for the user to choose from.

[0103] See Figure 6 This figure is a schematic diagram of an interactive page in a multi-select scenario proposed in an embodiment of this application. Figure 6As shown, for the key frame generation stage, 2 key frames are generated. When the content confidence of key frame 2 (the result of the initial stage) is low, 4 stage results can be generated: "four feet on the ground", "front feet land first", "land on the side", and "one foot land". Only options are schematically shown in the figure, and multiple stage results generated based on the initial stage result can be displayed in the preview display area. The user can select at least one stage result from the 4 stage results for subsequent video generation. Such as "four feet on the ground" and "front feet land first".

[0104] Thus, by generating L stage results when the content confidence of the initial stage result reaches the confidence condition relative to the video description text, it is possible to actively expand multiple stage results for the user to select when there is a risk of a large deviation in the initial stage result, thereby increasing the probability of obtaining a result that meets the user's expectations in an uncertain scenario, increasing the fault tolerance space for the user's selection, and improving the flexibility of video generation and the consistency between the target video and the user's expectations.

[0105] B21: In response to the feedback information of the effect evaluation problem being to accept Q of the L stage results, video generation for the (i + 1)-th generation stage is performed based on the Q stage results respectively, and other stage results in the L stage results except the Q stage results are deleted.

[0106] Among them, 1 ≤ Q < L. The user can select among the L stage results, select Q of the stage results for subsequent video generation, and delete other stage results that are not selected, thereby screening out the generation paths that meet the user's expectations among multiple branches.

[0107] Thus, by generating multiple stage results at the end of the i-th generation stage and allowing the user to select, the video generation process is extended to a multi-branch generation path. The user can compare and screen different stage results in the same generation stage, increasing the probability that the stage results match the user's expectations. By the user selecting Q stage results and only continuing the subsequent generation based on the selected stage results, it is avoided that branches that do not meet the expectations continue to participate in the subsequent generation process, thereby reducing the consumption of computing resources caused by ineffective generation, improving the generation efficiency, and further enhancing the controllability of the video generation process, and increasing the consistency between the target video and the user's expectations.

[0108] Trigger strategy two, uncertainty trigger.

[0109] In trigger strategy two, the interaction condition is the confidence condition for the content confidence. For the specific implementation of trigger strategy two, refer to C1 - C2, where C1 - C2 is a specific implementation of S4021 (i.e., the step of displaying the generated content and the evaluation question): C1: During the video generation process of the i-th generation stage, in response to the fact that the confidence level of the intermediate generated content of the i-th generation stage relative to the video description text reaches the confidence condition, it is determined that a content drift event has occurred in the intermediate generated content, and the video generation of the i-th generation stage is paused.

[0110] Intermediate generated content refers to the content generated during the video generation process. Content confidence describes the reliability of the generated content. Specifically, the content confidence of intermediate generated content relative to the video description text describes the degree of consistency between the intermediate generated content and the video description text. The higher the degree of consistency, the higher the content confidence. For example, if the video description text is "a basketball rolls from the basketball court to the sidelines," and the intermediate generated content shows a volleyball rolling, the volleyball and basketball are semantically inconsistent, indicating that the content confidence of the intermediate generated content relative to the video description text is low.

[0111] The confidence condition is an interactive condition used to determine whether a content drift event has occurred in the intermediate generated content. A content drift event is used to characterize a deviation between the intermediate generated content and the video description text. This application does not specifically limit the confidence condition. For example, if the content confidence level is lower than a content confidence level threshold, a content drift event is determined to have occurred. For instance, if the content confidence level ranges from 0 to 1, the content confidence level threshold can be 0.6. Furthermore, if, during video generation, the video generation model continuously generates corresponding intermediate generated content at each sampling time, and the content confidence level remains lower than the content confidence level for K consecutive sampling times, it indicates that the content confidence level has been persistently low, thus determining that a content drift event has occurred, thereby avoiding false triggering of content drift events due to instantaneous fluctuations. Additionally, if the proportion of content confidence levels lower than historical content confidence levels is greater than a proportion threshold, a content drift event is determined to have occurred.

[0112] In one possible implementation, different generation stages correspond to different confidence conditions, so that when determining whether a content drift event has occurred, the judgment is based on the confidence conditions that better match the current generation stage, thereby improving the accuracy of triggering and user interaction timing.

[0113] During the video generation process in the i-th generation stage, if the confidence level of the intermediate generated content in the i-th generation stage relative to the video description text reaches the confidence condition, indicating a significant deviation between the intermediate generated content and the video description text, it can be determined that a content drift event has occurred in the intermediate generated content. The video generation in the i-th generation stage is then paused, and the user is requested to intervene in the current video generation process to confirm and adjust the intermediate generated content.

[0114] The embodiments of this application do not specifically limit the calculation method of content confidence. The following describes four methods for calculating content confidence.

[0115] (1) Determine the content confidence based on the semantic similarity between the intermediate generated content and the video description text. Here, semantic similarity is used to characterize the degree of matching at the semantic level. The higher the semantic similarity, the higher the content confidence.

[0116] As one implementation method, semantic similarity can be obtained by inputting intermediate generated content and video description text into a Contrastive Language-Image Pre-training (CLIP) model.

[0117] (2) If the intermediate generated content includes multiple video frames, the content confidence can be determined by combining the temporal consistency score between adjacent video frames with the calculation method (1). The temporal consistency score is used to characterize the degree of continuity between adjacent video frames in the intermediate generated content. Under the condition that the semantic similarity remains unchanged, the higher the temporal consistency score, the higher the content confidence.

[0118] As one implementation method, the temporal consistency score can be determined based on the cosine similarity of adjacent video frames in the feature space. The smaller the difference in the cosine similarity between adjacent video frames, the higher the temporal consistency score.

[0119] (3) Based on the calculation method (1), the content confidence is determined by combining the normalized variance of the model prediction noise. The normalized variance of the model prediction noise is used to characterize the prediction uncertainty of the video generation model in the current generation step. In video generation models (such as diffusion models), the model usually needs to predict noise. The fluctuation of the prediction noise can be measured by the variance, and the normalized variance is obtained by normalization. The larger the normalized variance, the more unstable the prediction of the video generation model in the current generation step is, and the higher the uncertainty is; the smaller the normalized variance, the more stable the prediction of the video generation model is.

[0120] The higher the normalized variance of the model's prediction noise, the more uncertain the video generation model is about the current prediction, and the lower the content confidence.

[0121] (4) If the intermediate generated content includes multiple video frames, the content confidence can be determined by combining the temporal consistency score of adjacent video frames and the normalized variance of the model prediction noise based on the calculation method (1).

[0122] For example, calculation method (4) can be expressed by the following formula:

[0123] in, For content confidence threshold, For weights related to semantic similarity, For semantic similarity, For the weighting of the time-series consistency score, For time-series consistency score, The weights are for the normalized variance of the model's prediction noise. This is the normalized variance of the model's predicted noise. + + =1, for example, =0.4, =0.3, =0.3.

[0124] C2: Displays intermediate generated content, and displays drift evaluation questions built for content drift events and intermediate generated content.

[0125] A drift evaluation problem is a type of evaluation question used to guide users in evaluating content drift events corresponding to intermediate generated content. In this embodiment, the drift evaluation problem is constructed for both the content drift event and the intermediate generated content. Since the influencing factors that trigger content drift events can be diverse, drift evaluation problems can be generated and displayed for these influencing factors, allowing users to clearly identify which aspect of the video generation process has deviated.

[0126] After determining that a content drift event has occurred in the intermediate generated content, the intermediate generated content and drift evaluation issues can be displayed so that users can clearly understand which aspect of the currently generated content may have problems and deviates from the video description text.

[0127] As one approach, factors that trigger content drift events may include one or more combinations of semantic similarity, temporal consistency scores, or the normalized variance of model prediction noise.

[0128] For example, if the factor that triggers the content drift event is semantic similarity, it can display "The currently generated character setting is as follows, which may deviate from the video description text. Please confirm whether it meets your requirements."

[0129] The response to the drift evaluation problem is either to accept the intermediate generated content and continue video generation in the i-th generation stage based on the intermediate generated content, or to perform video generation in the (i+1)-th generation stage based on the intermediate generated content.

[0130] The feedback information in response to the drift evaluation problem is a content adjustment instruction for the intermediate generated content. Based on the content adjustment instruction, the video generation in the i-th generation stage is restarted.

[0131] Therefore, by dynamically judging whether content drift events occur based on content confidence during the video generation process, and actively pausing video generation when the content confidence reaches the confidence condition, the interaction trigger is more timely. By displaying drift evaluation issues, users have the opportunity to make timely adjustments when there may be deviations in the generated content, avoiding the continuous accumulation and amplification of deviations in subsequent generation processes. This improves the rationality of triggering user interaction timing, so that the target video is more consistent with user expectations.

[0132] Triggering strategy three: semantic change triggering.

[0133] In trigger strategy three, the interaction condition is a preset condition based on semantic changes in the content. For details of trigger strategy three, see D1-D3. D1-D2 are specific implementations of S4021 (i.e., the step of displaying the generated content and the question to be evaluated), and D3 is a specific implementation of S4022 (i.e., the step of continuing video generation). D1: During the video generation process of the i-th generation stage, in response to the semantic change of the intermediate generated content of the i-th generation stage reaching the preset condition, it is determined that a content switching event has occurred in the intermediate generated content, and the video generation of the i-th generation stage is paused.

[0134] Among them, semantic change describes the degree of change that occurs in the generated content at the semantic level. For example, changes in the scene of the target video (such as changes in the background of the video frame), characters entering or leaving the frame, changes in the type of action (such as changing from walking to running), and camera switching (such as switching from a close-up to a long shot) all fall under the category of semantic changes in the generated content.

[0135] The preset conditions are interactive conditions used to determine whether a content switching event has occurred in the intermediate generated content. A content switching event is used to characterize a significant semantic change in the intermediate generated content. This application does not specifically limit the preset conditions. For example, if the semantic change in content is greater than a change threshold, a content switching event is determined to have occurred. Taking a semantic change of 0-1 as an example, the change threshold could be 0.5.

[0136] During the video generation process in the i-th generation stage, if the semantic change of the intermediate generated content in the i-th generation stage reaches a preset condition, it indicates that the intermediate generated content has undergone a significant semantic change. In this case, it can be determined that a content switching event has occurred in the intermediate generated content, and the video generation in the i-th generation stage is paused to request the user to intervene in the current video generation process and confirm and adjust the intermediate generated content that has undergone semantic changes.

[0137] For example, the degree of semantic change used to characterize semantic changes in content can be calculated using the following formula:

[0138] in, The semantic change of the content at time t, The semantic feature vector extracted by the pre-trained visual encoder. Used to represent the video frame at time t. Used to represent A video frame at a given moment.

[0139] D2: Displays intermediate generated content, as well as switching evaluation questions for content switching events and intermediate generated content construction.

[0140] The switching evaluation question is a type of question to be evaluated, used to guide users to evaluate the content switching event corresponding to the intermediate generated content. In this embodiment, the switching evaluation question is constructed for both the content switching event and the intermediate generated content; that is, different types of semantic changes will cause different content switching times, thus displaying different switching evaluation questions.

[0141] After determining that a switching drift event has occurred in the intermediate generated content, the intermediate generated content and switching evaluation issues can be displayed, so that users can not only understand that the currently generated content has undergone significant semantic changes, but also understand what changes have occurred.

[0142] For example, if the semantic change of the content that triggers the content switching event is a change in the shot of an adjacent video frame from a close-up to a long shot, then the video frames corresponding to the close-up and long shot can be displayed respectively, and the message "Please check whether the content before and after the shot switching is as expected" can be displayed.

[0143] D3: In response to the feedback information of the switching evaluation question, accept the intermediate generated content and continue to generate video in the i-th generation stage based on the intermediate generated content.

[0144] After the user provides feedback on the switching evaluation issue, if the feedback message is to accept the intermediate generated content, it indicates that the intermediate generated content, which has undergone significant semantic changes, is largely consistent with the user's expectations. Therefore, the video generation in the i-th generation stage can continue based on the intermediate generated content.

[0145] If the feedback information for switching evaluation issues is a content adjustment instruction for the stage results, indicating that the intermediate generated content deviates from the user's expectations, then the video generation for the i-th generation stage can be restarted based on the content adjustment instruction.

[0146] For example, after displaying the video frames corresponding to the close-up and long shot respectively, and displaying "Please confirm whether the content before and after the shot change meets expectations", if the content adjustment instruction given by the user indicates that the shot in the long shot should not include character A, then the content adjustment instruction can be input into the video generation model, and the video frame corresponding to the long shot can be regenerated to generate a video frame that does not include character A.

[0147] Therefore, by triggering interaction based on semantic changes in the content during video generation, users are guided to confirm or adjust when significant semantic changes occur. This reduces the probability of generating unexpected content at key semantic turning points, allowing video generation to continue as expected even after significant semantic changes. This enhances the controllability of the video generation process and the consistency between the target video and user expectations at the semantic level.

[0148] Triggering strategy four: User-defined trigger.

[0149] In trigger strategy four, the video description text also includes a pause interaction condition, which is another interaction condition. For details on trigger strategy four, see E1-E3. E1-E2 are specific implementations of S4021 (i.e., the step of displaying generated content and the question to be evaluated), and E3 is a specific implementation of S4022 (i.e., the step of continuing video generation). E1: During the video generation process of the i-th generation stage, in response to the intermediate generated content of the i-th generation stage reaching the pause interaction condition, the video generation of the i-th generation stage is paused.

[0150] The pause interaction condition is an interaction condition that the user pre-indicates in the video description text, used to determine when to trigger the pause of video generation. For example, the pause interaction condition can be "pause when generating character appearance", "pause when a jumping action occurs", "pause when generating the Kth frame", etc.

[0151] During the video generation process in the i-th generation stage, if the intermediate generated content of the i-th generation stage reaches the pause generation condition, indicating that the user wants to pause while generating the intermediate generated content, then the video generation of the i-th generation stage can be paused, so that the user can intervene in the video generation process at the time when they want to pause generation.

[0152] As one implementation method, during the video generation process in the i-th generation stage, features can be extracted from the pause interaction conditions included in the video description text to obtain the corresponding semantic information. If the similarity between the intermediate generated content and the semantic information is greater than the similarity threshold, it is determined that the intermediate generated content has reached the pause interaction condition.

[0153] E2: Displays the intermediate generated content, as well as the pause evaluation questions built based on the pause interaction conditions and the intermediate generated content.

[0154] The pause evaluation question is a type of question to be evaluated, used to guide users to evaluate intermediate generated content that has reached the pause interaction condition. In this embodiment, the pause evaluation question is constructed for the intermediate generated content, that is, it asks questions about the intermediate generated content. For example, when the pause interaction condition is "pause while generating the cat's appearance", the currently generated cat's appearance can be displayed, along with "Does the current character appearance meet expectations?"

[0155] Once the conditions for pausing interaction are met by determining that the intermediate generated content has been met, the intermediate generated content and the paused evaluation question can be displayed so that users can provide feedback on the paused evaluation question.

[0156] E3: In response to the feedback information of the pause evaluation problem, the intermediate generated content is accepted, and video generation in the i-th generation stage is continued based on the intermediate generated content.

[0157] After the user provides feedback on the pause evaluation issue, the response message is "accept the intermediate generated content." This indicates that the user is relatively satisfied with the intermediate generated content generated when the pause interaction condition was met. In other words, the video generation process before the pause interaction condition was met was largely consistent with the user's expectations. Therefore, the video generation of the i-th generation stage can continue based on the intermediate generated content.

[0158] The feedback information in response to the pause evaluation issue is a content adjustment instruction for the stage results, indicating that the intermediate generated content deviates from the user's expectations. In this case, the video generation of the i-th generation stage can be restarted based on the content adjustment instruction.

[0159] Therefore, by introducing pause interaction conditions into the video description text, the interaction trigger can be actively defined by the user according to their own needs, thus making the timing of the interaction more closely match the generated content that the user is interested in. The ability to pause at specific points or when specific content that the user cares about allows the user to intervene in the video generation process at a time that better aligns with their expectations, improving the flexibility and personalization of the interaction.

[0160] This application does not specifically limit how the content adjustment instruction is processed to continue the subsequent video generation process. The following is an exemplary description of one implementation method, please refer to F1 for details: F1: In response to feedback on the issue to be evaluated, provide content adjustment instructions for the generated content and update the video description text accordingly.

[0161] The updated video description text is applied to video generation in the generation stages following the i-th generation stage out of the N generation stages.

[0162] Upon receiving a content adjustment instruction, the original video description text can be updated, such as by supplementing, replacing, or correcting it, to form an updated video description text. This allows subsequent generation stages to continue video generation based on the updated description text.

[0163] For example, when a user gives instructions for content adjustment in natural language (such as "the cat should be a little fatter", "walk a little slower", "change the background to a warmer color"), these instructions can be parsed into structured video description text by calling a large language model.

[0164] Parsing process: The user's text feedback, along with the current generation context (including video description text, information about the current generation stage, and a description of the generated content), is fed into the large language model. The large language model is then instructed to output the updated video description text, in the following format: { "target_element": "Character-Cat", "attribute": "body type", "operation": "increase", "magnitude": "moderate" "specific_description": "More rounded body shape, rounder face" } The updated video description text is then processed by the conditional dynamic injector of the video generation model for subsequent video generation.

[0165] Therefore, by updating the video description text after receiving content adjustment instructions, the user's modification intentions can be applied to the subsequent generation stage in the form of text constraints. This avoids the problem of local adjustments being made only in the current generation stage and becoming invalid in subsequent generation processes. By updating the video description text, the subsequent generation process is always based on the updated video description text, and the generated content can better match the user's modification intentions, improving the consistency of the target video.

[0166] This application does not specifically limit the form of the feedback information. For example, the feedback information can be in text form or in visual annotation form. The following describes a method for obtaining feedback information in the form of visual annotation, specifically referring to G1-G2, where G1 is a specific implementation of S4021: G1: Display the generated content and the evaluation questions for the generated content when the interaction conditions are met in the i-th generation stage through an interactive page.

[0167] The interactive page is used to interact with users during the video generation process. The interactive page can visually present the generated content to users by displaying the generated content and questions to be evaluated.

[0168] G2: Obtain feedback information for the issues to be evaluated through the interactive page.

[0169] The interactive page supports multiple input methods, including text input, option selection, and visual annotation. Users can input feedback information through the input methods provided on the interactive page.

[0170] The feedback information includes at least one of the following: content adjustment text, or content adjustment annotations on the generated content. Content adjustment text is a textual instruction for content adjustment, while content adjustment annotations are visual instructions for content adjustment, used to annotate the generated content displayed on the interactive page to indicate the scope of the adjustment.

[0171] See Figure 7 This figure is a schematic diagram of an interactive page for content adjustment annotation according to an embodiment of this application. Users can enter the text "The cat should be a little fatter" in the input box of the interactive page. Users can circle the cat's body area in keyframe 2 to instruct the regeneration of that body area.

[0172] As one implementation method, content adjustment instructions can be parsed into modification commands. The parsed modification commands are then transformed into conditional inputs acceptable to the video generation model through the conditional dynamic injector of the video generation model. The following are four examples illustrating conditional injection methods: Cue-level injection: Modifying or supplementing text cue words (a type of video description text). For example, changing the original cue word "an orange cat" to "a round-faced, chubby orange cat".

[0173] Parameter-level injection: Directly modify the control parameters of the video generation model. For example, adjust the motion speed parameter from speed=1 to speed=0.6.

[0174] Conditional graph-level injection: Generates or modifies reference images required for conditional control in systems like ControlNet. For example, it updates the human pose guidance graph based on user feedback.

[0175] Attention-level injection: By modifying the weight distribution of the attention layer of the video generation model, the video generation model is guided to focus on or ignore specific semantic elements and spatial regions.

[0176] For example, when a user inputs content adjustment annotations on an interactive page (such as selecting areas or drawing annotations), these content adjustment annotations can be converted into spatial condition constraints for subsequent video generation. The following three conversion methods are illustrated with examples: (a) The selected region is converted into a spatial attention mask and marked as a region that "needs to be regenerated"; (b) The graffiti content is transformed into ControlNet-style edge-guided or color-guided conditions; (c) The content outside the selected area remains unchanged, and only the selected area is locally regenerated (refer to Image Inpainting).

[0177] Therefore, by acquiring user feedback information in a multimodal manner through the interactive page, users can not only express their modification intentions through text, but also express more intuitive and precise adjustment intentions by directly annotating the generated content. This improves the correlation between feedback information and generated content, and enhances the adjustment accuracy of subsequent video generation.

[0178] In one possible implementation, to enable content adjustment instructions to apply not only to the current generation stage but also to historical generation stages, this application proposes a cross-stage retrospective adjustment mechanism based on target video elements, as detailed in H1-H2: H1: In response to the feedback information of the problem to be evaluated, the content adjustment instructions for the generated content are determined, and the target video elements involved in the content adjustment instructions are identified.

[0179] The target video element refers to the element involved in the content adjustment instruction. For example, the target video element may include characters, scenes, objects, actions, styles, or other elements that can constitute video content.

[0180] This application does not specifically limit how the target video elements involved in the content adjustment instruction are determined. For example, if the content adjustment instruction is content adjustment text, the target video elements involved in the content adjustment text can be determined by semantic parsing of the content adjustment text using natural language processing technology. As another example, if the content adjustment instruction is a content adjustment annotation, the target video elements can be determined based on the spatial location of the content adjustment annotation in the generated content and its corresponding annotation information.

[0181] H2: In response to the correlation between the target video element and the stage result of the j-th generation stage, the video generation of the j-th generation stage is restarted based on the content adjustment prompt.

[0182] Among them, "relevant" is used to represent that there is a generation dependency relationship between the target video element and the stage result of a certain generation stage. For example, when the target video element is generated or determined in a certain generation stage, it is determined that the target video element is relevant to the stage result of this generation stage. Another example is that the stage result of a certain generation stage is generated based on the target video element, and it can also be determined that the target video element is relevant to the stage result of this generation stage.

[0183] The j-th generation stage is the generation stage executed before the i-th generation stage, where 1 ≤ j < i. After determining the target video element, it is judged which historical generation stage's stage result the target video element is relevant to. If it is relevant, then trace back to the corresponding j-th generation stage, and re-perform the video generation of this generation stage based on the content adjustment instruction, so as to update this stage result.

[0184] The stage locking and incremental generation mechanism proposed in the embodiments of the present application will be introduced below.

[0185] In a possible implementation manner, when the user confirms the stage result of the current generation stage at a certain interaction checkpoint (that is, when the interaction condition is reached), the stage result of this generation stage can be stored in the locked result pool, and its status is marked as the locked state. The locked stage result remains immutable during the subsequent generation process, that is, the subsequent generation stage will not modify it, so as to ensure the stability of the confirmed content. At the same time, the locked stage result participates in the video generation as a constraint condition for the subsequent generation stage, so that the subsequent generated content is consistent with the confirmed stage result in semantics and structure, reducing the probability of generation deviation. In addition, when the preset condition is met, the user can also perform an unlocking operation on the locked stage result. After unlocking, it is necessary to re-perform the video generation for this generation stage and all its downstream generation stages, so as to support the backtracking adjustment of the generation process. On this basis, the embodiments of the present application also propose an incremental generation mechanism. Specifically, when the user gives a content adjustment instruction at a certain interaction checkpoint, first determine the scope affected by this adjustment. If it only involves the current generation stage, only locally re-generate the corresponding generated content of the current generation stage. If it involves the locked result of the upstream stage, it is necessary to first unlock the corresponding upstream stage, and sequentially re-generate this stage and its subsequent dependent stages, so as to ensure the consistency of the overall generated content. At the same time, before performing the re-generation, it is also possible to prompt the user the generation scope that will be affected by this modification, so that the user can confirm whether to execute it, so as to reduce unnecessary repeated calculations while ensuring the generation accuracy, improve the video generation efficiency, and thus improve the consistency between the target video and the user's expectations.

[0186] Therefore, by identifying the target video element involved after receiving a content adjustment instruction, and regenerating the corresponding generation stage based on the correlation between the target video element and the historical generation stage, it is possible to correct relevant content from the upstream stage of the generation chain, reducing the continuous propagation of deviations in subsequent generation processes. Furthermore, by regenerating only the generation stage related to the target video element, without repeating calculations across all generation stages, generation accuracy is improved while computational resource consumption is reduced.

[0187] To improve the quality of generated questions for evaluation and their matching degree with the generated content, this application proposes a priority-based question selection mechanism, as detailed in I1-I4: I1: Generate candidate questions to be evaluated corresponding to the generated content.

[0188] The candidate questions to be evaluated are multiple questions automatically generated for the generated content, designed to guide users in evaluating the content.

[0189] In one possible implementation, the candidate problem to be evaluated is a closed-ended structured problem.

[0190] I2: Based on the content elements involved in the candidate questions to be evaluated, determine the element confidence and element type weight of the content elements relative to the video description text.

[0191] Candidate questions for evaluation can involve at least one content element, which is a component of the generated content that can be evaluated individually. For example, "cat's size" is a character element, "nighttime background" is a scene element, "jumping action" is an action element, and "warm color tone" is a style element. Different candidate questions for evaluation can involve different content elements.

[0192] Feature confidence describes the degree of match between content features and video description text. The higher the degree of match between content features and video description text, the higher the feature confidence.

[0193] The element type weight represents the importance of a content element to the generated target video. For example, the element type weight of a character element is higher than that of a background element. For instance, the element type weight of a character's appearance is 1, the element type weight of a scene layout is 0.8, the element type weight of background details is 0.5, and the element type weight of lighting tone is 0.6.

[0194] For each content element, its element type can be determined to obtain the corresponding element type weight. The element confidence level is then determined by calculating the matching degree between the content element and the video description text.

[0195] For example, content elements and video description text can be input into a pre-trained semantic matching model to obtain the semantic similarity between the two, and the semantic similarity can be used as the element confidence score.

[0196] I3: Determine the display priority of candidate evaluation issues by considering feature confidence, feature type weight, and associated candidate evaluation issues.

[0197] Display priority reflects the priority of displaying candidate issues for evaluation. A lower element confidence indicates a greater deviation from the video description text, resulting in a higher display priority. A higher element type weight indicates a more important element type for generating the target video, also resulting in a higher display priority.

[0198] As one implementation method, the display priority can be calculated using the following formula:

[0199] in, Used to characterize the display priority of the i-th content element Let i be the element type weight of the i-th content element. Let be the element confidence score of the i-th content element.

[0200] I4: The R candidate problems with the highest priority will be displayed as the problems to be evaluated.

[0201] R is an integer greater than 1. It can display the single candidate question with the highest priority as the question to be evaluated, or it can display multiple candidate questions with the highest priority as the questions to be evaluated, thereby improving the relevance of the questions to be evaluated and allowing users to provide more comprehensive feedback on multiple questions involving multiple content elements.

[0202] Therefore, by prioritizing candidate questions based on the confidence level and type weight of content elements, questions corresponding to important content elements that deviate from user expectations are presented to users first. This improves the matching degree between the questions to be evaluated and the currently generated content, as well as the relevance of the questions. This allows users to more efficiently locate and adjust key issues in the generated content, thereby reducing the interaction burden and improving feedback efficiency.

[0203] This application does not specifically limit the form of the question to be evaluated. In one possible implementation, the question to be evaluated can be a closed-ended structured question. In this case, both the question content and answer options of the closed-ended structured question are in a structured form, used to guide the user to make selections within a defined range, thereby obtaining standardized feedback information.

[0204] For example, closed-ended structured information can include at least the following three forms: (1) Binary confirmation type: Confirmation is made for a single content element. Users can provide feedback by affirming or denying. For example, "Is this character image acceptable?", the answer options are "Yes" and "No".

[0205] (2) Multiple-choice preference type: Provides multiple candidate solutions for the user to choose from. The user can select a candidate solution through a selection button. For example, options corresponding to solution A, solution B and solution C are given for the user to select at least one candidate solution.

[0206] (3) Parameter adjustment type: Adjustment of a quantifiable content element. Users can provide feedback through sliders, numerical input, etc. For example, the interactive page provides a slider control for movement speed, and users can input movement speed by dragging the input slider.

[0207] Therefore, by using closed-ended structured questions as the evaluation questions, users can quickly make selections within a preset range of options, thereby reducing the difficulty for users to express feedback information and improving interaction efficiency. Since the answer options are predetermined, feedback information with a uniform format and easy parsing can be obtained, thereby reducing the difficulty of subsequent processing and improving the accuracy of recognizing user intent.

[0208] In one possible implementation, the question to be evaluated can also be open-ended, such as a freely modifiable question that accepts natural language modification instructions from the user, allowing the user to input feedback information in a text input box. For example, the question to be evaluated could be "Please describe how you would like to modify it."

[0209] To enable the display strategy of the problem to be evaluated to adaptively adjust according to user interaction, thereby improving interaction efficiency, this application proposes a problem display adjustment mechanism based on feedback quality, as detailed in J1: J1: Adjust the problem display parameters for the next problem to be evaluated in response to the quality of the feedback information obtained.

[0210] Feedback quality describes the quality of user feedback and includes at least one of the following: the interval between displaying the question to be evaluated and obtaining the corresponding feedback, or the richness of the feedback content. Richness reflects the amount of information provided in the feedback; higher richness indicates more detailed descriptions and more content. Lower richness indicates simpler feedback with limited content.

[0211] As one implementation method, the richness of feedback information includes the modification rate, which characterizes the proportion of feedback information that indicates content modification. The higher the proportion of feedback information that indicates content modification, the richer the content.

[0212] The problem display parameters are used to indicate how the problems to be evaluated are displayed. The problem display parameters include at least one of the following: the number of problems to be evaluated displayed, the display frequency, or the query granularity of the problems to be evaluated. The number of displays refers to the number of problems to be evaluated displayed in one interaction. The display frequency is used to characterize the frequency of occurrence of the problems to be evaluated during the video generation process. The query granularity is used to characterize the degree of semantic refinement of the problems to be evaluated.

[0213] Specifically, if the interval between displaying the question to be evaluated and obtaining the corresponding feedback is longer, it indicates that the user prefers quick confirmation. In this case, the system can enter a trust-based quick mode, reducing the number of displayed questions, the frequency of display, or lowering the granularity of the questions to be evaluated. Conversely, if the feedback information is richer in content, it indicates that the user is dissatisfied with the currently generated content. In this case, the number of displayed questions, the frequency of display, or the granularity of the questions to be evaluated can be increased.

[0214] For example, this can be achieved by maintaining the interaction density coefficient. Reflects the quality of feedback. For example, The value range can be [0.2, 2.0], with an initial value of 1.0. Interaction trigger threshold. Used to trigger interaction flows based on interaction density coefficients. Interaction trigger threshold. The interaction density coefficient is used to determine the frequency at which the problem to be evaluated is displayed. The larger the value, the more frequently the issues to be evaluated will be displayed.

[0215] The update rules are as follows: If a user quickly confirms all K consecutive interactions (e.g., the interval between displaying the question to be evaluated and receiving the corresponding feedback is less than a threshold, and the user selects to accept the generated content), then the user is considered to have entered the trust acceleration mode, and the interaction density coefficient is reduced. For example, the default value for K is 3, and the default value for the interval threshold is 2 seconds. For example, the interaction density coefficient can be reduced using the following formula. :

[0216] in, The updated interaction density coefficient, This is the interaction density coefficient before the update.

[0217] If the modification rate of the most recent K interactions is greater than the modification rate threshold (e.g., 0.6), it is determined that the user is not satisfied with the current generated content, and the interaction density coefficient is increased. For example, the interaction density coefficient can be increased using the following formula. :

[0218] in, The updated interaction density coefficient, This is the interaction density coefficient before the update.

[0219] Therefore, by dynamically adjusting the display parameters of the questions to be evaluated based on the quality of the feedback information, the display method of the questions to be evaluated can adapt to the user's interaction. This simplifies the interaction and improves the efficiency of interaction when the user quickly confirms the information. When the user is not satisfied with the generated content, more detailed or more questions to be evaluated are provided so that the user can express their modification intentions more comprehensively.

[0220] To reduce the computational resource consumption of content generation and display during interaction and to improve interaction response speed, this application proposes a low-resolution preview display method for generated content, as detailed in K1-K2: K1: Get the original generated content when the interaction conditions are met in the i-th generation stage.

[0221] The original generated content is the unprocessed generated content when the interaction conditions are met in the i-th generation stage. For example, the original generated content can be a high-resolution video frame, a detailed character model, or a scene image including complex lighting effects.

[0222] K2: By reducing the rendering quality of the original generated content, the generated content is previewed and displayed.

[0223] Rendering quality characterizes the level of detail in the visual presentation of generated content. It can be quantified by metrics such as resolution, texture detail, lighting effects, and anti-aliasing. Lowering rendering quality typically means reducing computational load, thereby speeding up display and enabling faster previews.

[0224] In one possible implementation, the rendering quality of the original generated content can be reduced in the following way: (1) Reduce resolution: Generate at 1 / 4 to 1 / 2 of the target resolution; (2) Reduce the number of denoising steps: reduce the number of denoising iterations of the diffusion model from a full T steps to T / 4 steps, sacrificing some detail quality in exchange for speed; (3) Reduce frame rate: For video clip previews, generate at 1 / 4 to 1 / 2 of the target frame rate; (4) Skeletal animation replacement: In the motion planning stage, simplified skeletal line animation is used instead of full video rendering to show the motion effect.

[0225] Reducing the rendering quality of the original generated content means controlling the waiting time for each piece of generated content to be displayed within a preset time range, such as 3-10 seconds, while ensuring that users can make accurate judgments.

[0226] Therefore, by down-rendering the original generated content during the interaction process and then previewing it, the generated content can be quickly presented to the user with lower computational overhead, thereby reducing the latency caused by high-quality rendering, improving the interaction response speed, and reducing the consumption of computing resources.

[0227] Furthermore, in response to the feedback information of the question to be evaluated, the generated content is accepted and the original generated content is displayed.

[0228] When a user evaluates and accepts the generated content displayed in the preview, it means that the current generated content meets the user's expectations in terms of effect. In this case, the low-rendering quality generated content used for the preview can be replaced with the corresponding original generated content for display.

[0229] As one implementation approach, once the user confirms satisfaction with the generated content displayed in the preview, formal rendering is performed at full resolution, denoising steps, and frame rate. Since the generation direction has been confirmed at this point, the result of full-quality rendering is highly consistent with the generated content displayed in the preview, with improvements only in image detail and quality.

[0230] Therefore, by displaying the original generated content only after the user confirms the generated content based on the preview, the generated content can be triggered on demand. This avoids high-overhead rendering of all generated content during the interaction phase, reducing the consumption of computing resources. This not only improves the response speed during the interaction phase, but also ensures that the final display effect of the confirmed generated content is sufficiently refined. It improves interaction efficiency while taking into account the generation quality, thus enhancing the user experience.

[0231] In one possible implementation, this application proposes an adaptive training model feedback mechanism, as detailed in L1-L2: L1: Construct training samples based on feedback information.

[0232] Each feedback message regarding the question to be evaluated is a valuable signal of user preference for the machine, such as the user's confirmation / modification choice, modification direction, and preferred candidate solutions. This data can be used to construct training samples, for example: (a) Build a user preference database to achieve personalized video generation (understand what style, color tone, and movement rhythm a specific user prefers). (b) As training samples for Reinforcement Learning from Human Feedback (RLHF), the quality of the underlying video generation model is continuously optimized so that the model gradually learns "which generated content is likely to be confirmed by human users", thereby improving the hit rate and reducing the need for future interactions. (c) These serve as training samples for the question generation model, making the questions increasingly precise and better able to address the key concerns of users. The question generation model is used to generate questions to be evaluated.

[0233] L2: Optimizes the video generation model used to generate the target video and the question generation model used to generate the question to be evaluated using training samples.

[0234] The embodiments of this application do not specifically limit how to optimize the video generation model and the question generation model.

[0235] As one implementation approach, for video generation models, supervised learning training can be performed based on training samples. This involves comparing the generated content with the user's training samples to construct a loss function and update the model parameters. Alternatively, preference pairs can be constructed based on user-confirmed generated content or preference choices, and the video generation model can be trained using reinforcement learning based on user feedback to make the generated content more aligned with user preferences.

[0236] As one implementation approach, the probability of generating different evaluation questions can be adjusted based on user feedback (such as whether an answer was given, response time, and the content of the feedback). Alternatively, the question generation model can be optimized based on the content elements involved in user feedback, making it more likely to generate evaluation questions related to the content elements that users are interested in.

[0237] Therefore, by transforming user feedback during the interaction process into training samples, and optimizing the video generation model and question generation model based on these training samples, the model can be adaptively optimized using real user feedback. This gradually improves the quality of video generation and the accuracy of the questions to be evaluated, forming a positive cycle in which users provide feedback, the feedback is used to optimize the model, the model generation quality is improved and user satisfaction is further enhanced, thereby attracting more users to use the system.

[0238] To enhance user control over the overall video generation process, this application proposes a historical generation stage regeneration mechanism based on stage backtracking, as detailed in M1-M2: M1: In response to the stage backtracking operation, displays the historical generation stages for which video generation has been completed for the target video.

[0239] The stage rewind operation is a user-initiated action used to rewind to a previous generation stage. For example, if the interactive page displays multiple generation stages, the user can initiate a stage rewind operation by dragging the progress bar corresponding to each generation stage. The historical generation stage refers to the generation stage where video generation has already been completed.

[0240] M2: Obtain the first content adjustment instruction for the first historical generation stage in the historical generation stage, and regenerate the video for the first historical generation stage and subsequent historical generation stages based on the first content adjustment instruction.

[0241] The first historical generation stage is a single historical generation stage, and the first content adjustment instruction is a content adjustment instruction for the first historical generation stage. Subsequent historical generation stages are those that are executed later than the first historical generation stage.

[0242] In other words, taking the first historical generation stage as an example, when a user gives the first content adjustment instruction for the first historical generation stage, the generation stage and subsequent completed historical generation stages are regenerated, starting from the first historical generation stage, thereby updating the entire video generation chain.

[0243] For example, if the historical generation stages are, in order of execution, the character setting stage, the keyframe stage, and the short video generation stage, and the user reverts to the character setting stage through a stage backtracking operation and requests "the cat should be a little fatter," then based on this content adjustment instruction, the cat's appearance generation in the character setting stage, keyframe stage, and short video generation stage will be performed again.

[0244] Therefore, by displaying historical generation stages in the stage backtracking operation and regenerating that historical generation stage and its subsequent historical generation stages based on the user's instructions to adjust the content of a specified historical generation stage, users can make global adjustments to the video generation process, rather than being limited to the current generation stage. Users can not only correct relevant content from a completed node, thus avoiding the continuous impact of previous stage deviations on subsequent generated content, but also, by regenerating all relevant subsequent stages from the first historical generation stage, the regeneration process is automated without requiring multiple user instructions, improving the efficiency of the interactive system and lowering the operational threshold.

[0245] This application does not specifically limit the method of dividing the generation stages. For example, video generation can be divided into stages according to the time dimension, dividing the video generation into multiple time segments of equal length, with each time segment corresponding to a generation stage. Alternatively, the stages can be divided according to the processing order of the video generation model, dividing video generation into text parsing, conditional encoding, content generation, and post-processing stages. Furthermore, the stages can be divided according to the spatial regions of the video content, defining the generation process corresponding to different spatial regions as different generation stages. To ensure that the division of generation stages can both cover the overall structural design of the video and adapt to the dynamic changes of video content in the time dimension, this application proposes a dynamic and static combined stage division method, as detailed in N1-N2: N1: The video generation process is divided into stages, resulting in M ​​generation stages involving the global static design of the video and P generation stages involving the local dynamic content of the video.

[0246] The video's global static design describes content that defines the overall structure of the video and does not change frequently over time. For example, the scenes and characters in the video constitute the main framework and do not undergo significant changes. The video's local dynamic content describes content that changes over time, such as character movement trajectories and camera transitions. 1 ≤ M, 1 ≤ P, M + P = N.

[0247] In one possible implementation, semantic parsing of the video description text can be performed to extract the relevant content elements. These elements are then categorized based on whether they change significantly over time. Content elements that remain stable throughout the entire video's temporal sequence are defined as the global static design, while those that change over time are defined as local dynamic content. Furthermore, the video generation process is divided into stages based on this classification. The generation task involving the global static design is divided into M stages to construct the overall structure and basic semantic framework of the video. The generation task involving local dynamic content is divided into P stages to generate the dynamic content that evolves over time. For local dynamic content, further subdivisions can be made into multiple generation stages corresponding to consecutive time segments, resulting in a generation stage division that simultaneously covers both the overall video structure and the dynamic change process.

[0248] N2: Determine the execution order of the N generation stages based on the execution dependencies between the M and P generation stages.

[0249] Among them, the execution order of the M generation stages is earlier than that of the P generation stages.

[0250] After dividing the generation process into stages, the execution order can be determined based on the dependencies between each stage. For example, a large language model can be used to analyze the input-output relationships between each stage, using the preceding generated content that subsequent stages depend on as the basis for dependency, thus constructing a dependency chain between the stages. For instance, M generation stages involving global static design of the video are used to determine the overall structure and basic semantic constraints of the video; their generated content serves as a unified input condition for subsequent generation and can be executed first. P generation stages involving local dynamic content of the video depend on the generated content of the global static design and are generated step by step. Therefore, they are executed sequentially after the M generation stages are completed, resulting in the execution order of N generation stages that satisfy the dependencies, ensuring that the video generation process proceeds in the order of first determining the global static content and then generating the local dynamic content. Furthermore, the execution order of each generation stage within the M stages and each generation stage within the P stages can also be determined in the above manner to determine the execution order of the N generation stages.

[0251] Therefore, by dividing the video generation stage into a generation stage involving the global static design of the video and a generation stage involving the local dynamic content of the video, and determining the execution order of the generation stages based on the execution dependency between the two, the video generation process proceeds in the order of first building the global structure and then generating the local dynamic content. This ensures that subsequent dynamic generation is always based on a unified and stable global structure, thereby reducing semantic conflicts between different generation stages and preventing subsequent dynamic generation from deviating from the user's expectations due to the instability of the preceding structure, thus improving the consistency between the target video and the user's expectations.

[0252] In one possible implementation, the N generation stages are based on the execution order, namely the global planning stage, keyframe generation stage, motion planning stage, fragment rendering stage, and stitching optimization stage. Each of these generation stages is described below.

[0253] (1) The global planning stage is used to generate the character settings, scene settings and storyboard settings for the target video.

[0254] During the overall planning phase, the target video can be planned as a whole based on user input.

[0255] For example, user input may include video description text, reference images, and style preferences.

[0256] After obtaining user input, a multimodal large language model (or a video generation model with the same capabilities) can be used to perform semantic analysis on the user input, thereby extracting entity information for representing the main body of the video, action events for representing the dynamic process of the video, and style attributes for representing the overall performance of the video.

[0257] Based on the extraction of the above content elements, generated content can be used to characterize the overall structure of the target video. For example, character design diagrams, scene design diagrams, and storyboards can be generated. Among them, character design diagrams are used to describe the appearance design of each character, scene design diagrams are used to describe the style and layout of the main scenes, and storyboards are used to divide the target video into multiple shots and label each shot with corresponding content descriptions, durations, and composition methods.

[0258] Furthermore, during the overall planning phase, users can interactively adjust the generated content. For example, adjustments can be made to character appearance, scene style, number of scenes, or scene order, making the overall planning result more in line with user expectations.

[0259] Therefore, by performing global planning on the target video during the global planning stage and generating character settings, scene settings, and storyboard settings, subsequent generation stages can generate video under a unified structural framework, thereby improving the overall consistency and controllability of the video generation process.

[0260] (2) The keyframe generation stage is used to generate the keyframe sequence set for the storyboard corresponding to the target video.

[0261] The keyframe generation stage, based on the content description of each shot in the storyboard, generates representative static images for each shot in the time dimension, thus serving as the basis for the generation of dynamic content in the subsequent video generation process. By generating keyframe sequences, the core images of the video at each time point can be predefined, ensuring semantic consistency and structural continuity of the generated content as time progresses.

[0262] For example, during the keyframe generation phase, keyframes can be generated based on the character settings, scene settings, and storyboard settings generated during the global planning phase. The character settings, scene settings, and storyboards can be used as input information to guide the keyframe generation process, thereby ensuring that the generated keyframes conform to the defined global semantic constraints.

[0263] For each scene in the storyboard, a corresponding keyframe is generated. For example, an image generation model (or a video description model with similar capabilities) can be used to process the content description of each scene to generate at least one keyframe, thus forming a keyframe sequence. The image generation model can be implemented based on deep learning and used to generate corresponding image content based on the input semantic information.

[0264] During keyframe generation, consistency constraints can be imposed on the generated content. For example, character settings can constrain the appearance of characters in keyframes, ensuring consistency across different keyframes. Similarly, scene settings can constrain the background style and spatial structure of keyframes, maintaining consistency across different keyframes and thus improving the overall consistency of the keyframe sequence.

[0265] The output of the keyframe generation stage can be a keyframe sequence. The keyframe sequence is used to represent the core frames of the target video at the corresponding time points of each shot, and serves as the input basis for subsequent generation stages (such as motion planning stage or segment rendering stage).

[0266] During the keyframe generation stage, users can also interactively adjust keyframes. For example, adjustments can be made to the composition, character pose, and image details of keyframes to better meet user expectations and further improve the accuracy of subsequent content generation.

[0267] (3) The motion planning stage is used to generate motion planning information for the key frame sequence and storyboard settings corresponding to the target video.

[0268] During the motion planning stage, the changes of various content elements in the video over time can be planned based on the keyframe sequence and the information describing the actions in the storyboard settings. This generates corresponding motion planning information to describe the movement of elements over time, thereby connecting discrete keyframes into a continuous dynamic process.

[0269] For example, in the motion planning phase, the transition process between adjacent keyframes can be modeled using the confirmed keyframe sequence and the motion description in the storyboard as input, thereby generating motion planning information.

[0270] Motion prediction models (such as those based on optical flow estimation or skeletal animation prediction, or video generation models with similar capabilities) can be used to plan the motion trajectories of various content elements between adjacent keyframes to determine how these elements change over time. The generated motion planning information can include the character's displacement path between frames, the camera's trajectory in space, and the timing of various actions, thus describing the dynamic process of the video from both spatial and temporal dimensions.

[0271] The motion planning stage can output corresponding motion trajectory data or motion preview results for use in subsequent generation stages. Furthermore, it allows users to adjust motion planning information, such as motion speed, motion path, or camera movement, to make the video's dynamic effects more consistent with expectations.

[0272] (4) The segment rendering stage is used to generate video segments corresponding to the storyboard settings of the target video.

[0273] The segment rendering stage can generate video content segment by segment in the time dimension based on keyframe sequences and motion planning information, thereby obtaining continuous video segments and providing a foundation for subsequent splicing to form a complete target video.

[0274] For example, in the segment rendering stage, keyframe sequences, motion planning information, character settings, and scene settings can be used as inputs to render and generate video content.

[0275] The target video can be divided into multiple video segments in chronological order, and each video segment can be rendered independently. For each video segment, the corresponding keyframes can be used as start and end constraints, and motion planning information can be used as dynamic guidance in the generation process, thereby generating video content that conforms to the semantics of the keyframes and satisfies the laws of motion change.

[0276] Furthermore, after each video segment is generated, it can be previewed to confirm the rendering result. Additionally, adjustments to the rendering results of the video segments are supported, such as optimizing visual effects, detail quality, or motion smoothness, thereby improving the overall quality of the generated video.

[0277] (5) The splicing optimization stage is used to obtain the target video by splicing video segments.

[0278] The stitching optimization stage can be based on the generated video segments, stitching the individual video segments together and optimizing the overall stitched video to obtain a continuous and visually consistent target video.

[0279] For example, in the stitching optimization stage, all confirmed video segments can be used as input, and the video segments can be stitched together in chronological order to form a complete video.

[0280] During the splicing process, the transition areas between adjacent video segments can be smoothed to reduce abrupt changes or discontinuities at the splicing points, thereby improving the coherence of the video.

[0281] Furthermore, global consistency optimization can be performed on the stitched video, such as uniformly adjusting the video's color and lighting to maintain a consistent overall visual style. Additionally, image quality enhancement processing can be applied to the stitched target video, for example, by increasing the resolution to improve video clarity, thereby further enhancing the display effect of the target video.

[0282] Therefore, by refining the video generation process into a global planning stage, a keyframe generation stage, a motion planning stage, a segment rendering stage, and a stitching optimization stage, and processing them in the order of execution, different types of content, such as characters, scenes, temporal evolution, and image details, can be handled at different generation stages. This reduces the coupling between various processing links and improves the stability and controllability of the generation process. Furthermore, users can intervene at each generation stage when interactive conditions are met. For example, adjusting characters, scenes, and storyboards in the global planning stage; optimizing composition and character poses in the keyframe generation stage; correcting motion paths in the motion planning stage; adjusting visual effects and detail quality in the segment rendering stage; and confirming overall style consistency in the stitching optimization stage. This allows different types of problems to be addressed specifically at their corresponding generation stages, reducing efficiency losses from repeated modifications, improving adjustment accuracy while reducing unnecessary regeneration operations, increasing video generation efficiency, and improving the consistency between the target video and user expectations.

[0283] See Figure 8 The figure is a schematic diagram of a system architecture proposed in an embodiment of this application. The system architecture includes a user interaction front-end, an interaction scheduling middleware layer, and a staged video generation engine.

[0284] The user interaction front-end includes an input panel, a preview area, and an interactive question and feedback area. The input panel receives user input (such as video description text, reference images, etc.). The preview area displays the generated video content. The interactive question and feedback area collects user feedback. The user interaction front-end can interact with the interaction scheduling middleware layer. For example, the user interaction front-end can communicate with the interaction scheduling middleware layer through an Application Programming Interface (API) or the WebSocket protocol.

[0285] The interactive scheduling middleware layer comprises four functional modules: a checkpoint scheduler, which has built-in interaction conditions and determines corresponding checkpoints based on these conditions, allowing the system to decide when to pause video generation and initiate interaction; an intelligent question generator, used to generate structured questions to be evaluated; a feedback parser, used to convert user feedback information into structured operation instructions; and a conditional dynamic injector, used to convert operation instructions into model condition constraints. The interactive scheduling middleware layer can interact with the staged video generation engine.

[0286] The staged video generation engine is used to actually generate target videos. The staged video engine can include a video generation model. The video generation process is divided into a global planning stage (executed by a global planner), a keyframe generation stage (executed by a keyframe generator), a motion planning stage (executed by a motion predictor), a segment rendering stage (executed by a segment renderer), and a stitching optimization stage (executed by a stitcher). Each generation stage can accept conditional constraints injected by an interactive scheduling intermediate layer.

[0287] Based on the above system architecture, when a certain checkpoint is triggered, i.e. when the interaction conditions are met, the following process can be executed.

[0288] Once a checkpoint is triggered, i.e., when the interaction conditions are met, the following process can be executed. See also... Figure 9 The figure is a schematic diagram of an interactive checkpoint-based process proposed in an embodiment of this application.

[0289] During the video generation process in the i-th generation stage, in response to the achievement of interactive conditions, the corresponding checkpoint (also known as the interaction checkpoint) is triggered, initially pausing the generation process of the staged video generation engine. After pausing, the corresponding intermediate generated content is extracted, and preview content for quick display is generated based on this intermediate generated content. Furthermore, the intermediate generated content can be lightweighted by reducing rendering quality or resolution, thereby generating a fast preview to improve interactive response efficiency.

[0290] After receiving the preview content, an intelligent question generator analyzes the generated content, extracts key content elements, and constructs corresponding evaluation questions based on these elements. The preview content and the corresponding evaluation questions are then presented to the user, allowing them to make judgments and provide feedback based on the generated content.

[0291] After receiving user feedback, different processing paths are initiated based on the type of feedback: If the feedback indicates acceptance of generated content, the current generation stage's result is locked, and subsequent generation stages continue based on this result. If the feedback indicates adjustments to the generated content, it is parsed, and the parsing results are dynamically injected into the staged video generation engine as constraints, causing a partial regeneration of the current generation stage's content. After partial regeneration, the corresponding preview content can be generated again, returning to the preview display stage, thus forming a closed-loop interactive process.

[0292] Therefore, by triggering the above process at interactive checkpoints, the video generation process is transformed from continuous execution into an interruptible and feedback-optimized process, allowing users to participate in decision-making during the generation process, thereby improving generation efficiency and the consistency between the generated content and user expectations.

[0293] The complete video generation process is described below using a division of generation stages. See also... Figure 10 The figure is a schematic diagram of a complete video generation process proposed in an embodiment of this application.

[0294] First, the user inputs video description text to describe the target video, expressing their requirements for generating the target video.

[0295] Subsequently, a global planning phase is generated based on the video description text, used to generate character settings, scene settings, and storyboard settings for the target video. After the global planning phase is completed, the process proceeds to interaction checkpoint 1. At interaction checkpoint 1, if the user confirms the currently generated content, the phase result of the global planning phase is locked, and the process enters the keyframe generation phase. If the user requests modifications to the currently generated content, the feedback information is parsed and processed, and the parsed results are injected as conditional constraints into the video generation model. The global planning phase is then regenerated until the user's expectations are met.

[0296] In the keyframe generation phase, a sequence of keyframes for the corresponding storyboard is generated based on the phase results of the global planning phase. After completing the keyframe generation phase, the process proceeds to interaction checkpoint 2. At interaction checkpoint 2, if the user confirms the keyframe generation result, the phase result of the keyframe generation phase is locked, and the motion planning phase begins. If the user requests modifications, the feedback information is parsed, conditions are dynamically injected, and the keyframe generation phase is regenerated.

[0297] In the motion planning phase, corresponding motion planning information, including character movement trajectory, camera movement path, and motion timing, is generated based on the keyframe sequence and storyboard settings. After completing the motion planning phase, the process proceeds to interaction checkpoint 3. At interaction checkpoint 3, if the user confirms the motion planning result, the phase result of the motion planning phase is locked, and the segment rendering phase begins. If the user requests modifications, the feedback information is parsed and conditions are injected, and the motion planning phase is regenerated.

[0298] During the segment rendering stage, corresponding video segments are generated based on keyframes and motion planning information. After completing the segment rendering stage, the process proceeds to interactive checkpoint 4. At interactive checkpoint 4, if the user confirms the segment rendering result, the stage result of the segment rendering stage is locked, and the process enters the stitching optimization stage. If the user requests modifications, the feedback information is parsed and conditional injection is performed, and the segment rendering stage is regenerated.

[0299] Finally, in the stitching optimization stage, the video segments are stitched together and global consistency optimization is performed to obtain the final target video and output it.

[0300] Therefore, by setting interactive checkpoints between each generation stage, users can confirm or adjust different types of generated content, such as global planning, keyframe generation, motion planning, and segment rendering, at each stage. This transforms the video generation process from a one-time generation to a multi-stage, step-by-step optimization process. While improving the controllability of the generation process, it reduces unnecessary repetitive generation operations, lowers resource consumption, improves interactive efficiency, and enhances the consistency between the target video and the user's expectations.

[0301] The following is through Figures 11-18 The timing flow diagram of the interactive checkpoints shown is used to illustrate the video generation process mentioned in the above embodiments.

[0302] See Figure 11 This figure is one of the timing flow diagrams of the interactive checkpoint proposed in an embodiment of this application. First, in step 1, the user inputs video description text through the input panel. For example... Figure 11 As shown, the user inputs the video description text "A cat jumps from a table to the ground," and can further set style preferences, such as realistic style, warm color tone, and natural lighting effects. After receiving the input, the system responds to the received video description text and begins executing Phase 1: the global planning phase.

[0303] Subsequently, in step 2, video generation is performed during the global planning phase based on the video description text. This generation is used to create character settings, scene settings, and storyboard settings for the target video. After completing the global planning phase, the system proceeds to interaction checkpoint 1. For example, the system can utilize a Multimodal Large Language Model (MLLM) to analyze the video description text, extract key entities and action events, and generate character setting diagrams, scene setting diagrams, and storyboards.

[0304] See Figure 12This diagram illustrates the second timeline of the interactive checkpoints proposed in this application. At interactive checkpoint 1, if the user confirms the intermediate generated content (or stage result), the stage result of the global planning stage is locked, and the keyframe generation stage begins. If the user requests modifications to the currently generated content, the feedback information is parsed and processed, and the parsing result is injected as a conditional constraint into the staged video generation engine to regenerate the global planning stage until the user's expectations are met. For example, the evaluation question Q1 is "The cat's appearance is like this (with preview image), are you satisfied?" The user answers "Needs modification." The evaluation question Q2 is "The indoor scene is in this style (with preview image), is it acceptable?" The user can answer "Needs modification" for evaluation question Q1 and "Acceptable" for evaluation question Q2.

[0305] See Figure 13 This diagram is the third in a series of sequential flow diagrams of the interactive checkpoints proposed in this application embodiment. The user replies to the question Q1, "The cat is a little fatter, and its face is rounder." The feedback parser first performs word segmentation on the feedback information, then updates the prompt words (video description text in this embodiment), resulting in "{target:"character-cat",attribute:"body shape",operation:"increase",specific:"rounder body shape, rounder face"} → prompt word update: "an orange cat" → "a round-faced, chubby orange cat" → only regenerate the character concept map, the scene setting (confirmed) remains unchanged." Then, the character concept map is regenerated and displayed again for user confirmation. After the user's second confirmation, the stage result of stage 1 is locked.

[0306] See Figure 14 This figure is the fourth of the timing flow diagrams for the interaction checkpoints proposed in this application embodiment. In step 3, the system executes stage 2: the keyframe generation stage. Specifically, based on the locked character settings (e.g., "a round-faced, chubby orange cat") and scene settings (indoor table scene), the system generates corresponding keyframes for each scene in the storyboard. For example, a keyframe sequence is generated for 4 scenes, including keyframe 1 (on the table), keyframe 2 (jumping), and keyframe 3 (landing). At the same time, the system can calculate the corresponding content confidence for each keyframe, for example, keyframe 1 is 0.92, keyframe 2 is 0.88, and keyframe 3 is 0.45. The content confidence of keyframe 3 is lower than the confidence threshold (e.g., 0.60), indicating that the keyframe has semantic deviation or unstable generation.

[0307] Subsequently, the system enters interactive checkpoint 2, the keyframe confirmation stage. Video generation is paused due to "uncertainty trigger + fixed stage trigger," meaning that the confidence level of keyframe 3 is below the threshold and it is at the stage boundary. The system freezes the generation state, extracts intermediate generated content, and generates a quick preview. Simultaneously, it analyzes the keyframe using an intelligent question generator and generates targeted evaluation questions based on element confidence and element type weights. For example, for the landing posture of keyframe 3, a multi-choice preference question is generated: "The content confidence level of the cat's landing posture is low; the system provides three candidate solutions," including "landing on all fours," "landing on its front paws first," "landing sideways," and "keep all, branch generation." A binary confirmation question is also generated: "You previously requested that the cat's face be rounder; does the current keyframe meet this requirement?" Options of "meets / still needs adjustment" are provided. The user selects solution A (landing on all fours) for the multi-choice question Q1 and "meets" for the binary question Q2.

[0308] See Figure 15 This figure is the fifth schematic diagram of the timing flow of the interaction checkpoint proposed in this application embodiment. After receiving the user's feedback information, the system only performs partial regeneration processing on keyframe 3, while keyframes 1 and 2 remain unchanged. After regeneration, the content confidence of keyframe 3 increases from 0.45 to 0.91, indicating that the generated content is significantly optimized. Subsequently, the system locks the generation result of stage 2 and writes the keyframe sequence into the locked result pool. At the same time, the system records interaction data, such as a user response time of 7 seconds and a modification rate of 1 / 2 (50%). This interaction data can be used for adaptive adjustment of subsequent interaction strategies.

[0309] See Figure 16 This figure is the sixth schematic diagram of the timing flow of the interactive checkpoint proposed in the embodiments of this application. In step 4, the system executes stage 3: motion planning stage. Specifically, based on the locked keyframe sequence, the system plans the motion trajectory of the character between each keyframe and determines the motion path and motion sequence of the camera. A skeletal animation preview is generated for user confirmation.

[0310] Next, the system enters interactive checkpoint 3, the motion preview confirmation stage. The generation process is paused due to a fixed stage trigger (the boundary between stage 3 and stage 4). The system displays the question, "Is the cat's walking speed appropriate? Please adjust the movement speed," and provides continuous adjustment controls (slider). The user can adjust the movement speed, for example, from 1.0x to 0.65x, and provides feedback such as, "A little slower, it feels more leisurely." The system adjusts and updates the motion planning information based on this parameter and regenerates the preview animation. After the user confirms, "This speed is acceptable," the system locks the stage 3 result.

[0311] See Figure 17 This figure is the seventh schematic diagram of the timing flow of the interactive checkpoint proposed in the embodiments of this application. In step 5, the system executes stage 4: segment rendering stage. Specifically, the system generates video content segment by segment based on the locked keyframe sequence and motion planning information. For example, the video is divided into multiple video segments (video segment 1, video segment 2, video segment 3), and each video segment is rendered with high quality to make it conform to the semantic constraints of keyframes and motion trajectory constraints.

[0312] Subsequently, the system proceeds to interactive checkpoint 4, the video clip confirmation stage. The generation process is paused due to a fixed stage trigger (the boundary between stages 4 and 5). The system displays the currently generated video clip and poses a binary confirmation question: "Does the currently generated video clip meet expectations?", providing the options of "Meets / Still Needs Adjustment". The user selects "Meets", indicating that the current clip quality meets the requirements, and the system locks the stage 4 result.

[0313] See Figure 18 This figure is the eighth of the timing flow diagrams for the interactive checkpoints proposed in this application embodiment. In step 6, the system executes stage 5: the stitching optimization stage. Specifically, the system stitches all locked video segments together to generate a complete video, and can also perform transition smoothing, color consistency correction, and optional super-resolution enhancement to improve the overall visual effect.

[0314] Finally, in the output stage, the system outputs the target video. For example, the generated video has a resolution of 1920×1080, a frame rate of 24fps, and a total duration of 8 seconds. Simultaneously, the system outputs interaction statistics, such as the 4 interaction checkpoints, 5 questions asked, 1 structural modification (character appearance adjustment), and 1 parameter adjustment (motion speed adjustment) made by the user during the generation process. Compared to traditional one-time generation methods that typically require 5-8 retries, this embodiment achieves one-time generation through a phased interaction mechanism, thereby significantly improving video generation efficiency and reducing user interaction costs.

[0315] In relation to the video generation method described above, this application also provides a corresponding video generation apparatus so that the above video generation method can be applied and implemented in practice.

[0316] See Figure 19 This figure is a schematic diagram of the structure of a video generation device provided in an embodiment of this application. Figure 19 As shown, the video generation apparatus 1900 includes an acquisition unit 1901 and a generation unit 1902; The acquisition unit 1901 is used to acquire video description text used to describe the target video; The generation unit 1902 is used to sequentially generate video for the target video in N generation stages according to the video description text, where N>1; The generation unit 1902 is further configured to obtain the target video through video generation in the Nth generation stage; In the video generation process involving the N generation stages, the generation unit includes a display unit 19021, a continued generation unit 19022, and a regeneration unit 19023. The display unit 19021 is configured to, when video generation in the i-th generation stage is performed, display the generated content of the i-th generation stage at the time the interaction condition is met, and the evaluation question for the generated content, 1≤i, in response to the achievement of the interaction condition. <N; The continuing generation unit 19022 is configured to respond to the feedback information of the problem to be evaluated as accepting the generated content, and continue video generation in the i-th generation stage based on the generated content, or perform video generation in the (i+1)-th generation stage based on the generated content; The regeneration unit 19023 is configured to, in response to feedback information regarding the problem to be evaluated being a content adjustment instruction for the generated content, regenerate the video in the i-th generation stage based on the content adjustment instruction.

[0317] As one possible implementation, the interaction condition is to complete the video generation of the i-th generation stage, and the display unit 19021 is specifically used for: Display the stage results output by the i-th generation stage and the effect evaluation question for the stage results; The continuing generation unit 19022 is specifically used for: The feedback information in response to the effect evaluation question is to accept the stage results and generate video for the (i+1)th generation stage based on the stage results.

[0318] As one possible implementation, the interaction condition is a confidence condition based on content confidence, and the display unit 19021 is specifically used for: During the video generation process of the i-th generation stage, in response to the fact that the content confidence of the intermediate generated content of the i-th generation stage relative to the video description text reaches the confidence condition, it is determined that the intermediate generated content has a content drift event, and the video generation of the i-th generation stage is paused. Display the intermediate generated content, and display the drift evaluation question constructed for the content drift event and the intermediate generated content.

[0319] As one possible implementation, the interaction conditions are preset conditions for changes in content semantics, and the display unit 19021 is specifically used for: During the video generation process of the i-th generation stage, in response to the semantic change of the intermediate generated content of the i-th generation stage reaching a preset condition, it is determined that a content switching event has occurred in the intermediate generated content, and the video generation of the i-th generation stage is paused. Display the intermediate generated content, and display the switching evaluation question constructed for the content switching event and the intermediate generated content; The continuing generation unit 19022 is specifically used for: In response to the feedback information of the switching evaluation problem, the intermediate generated content is accepted, and video generation in the i-th generation stage continues based on the intermediate generated content.

[0320] As one possible implementation, the video description text also includes a pause interaction condition, the interaction condition including the pause interaction condition, and the display unit 19021 is specifically used for: During the video generation process of the i-th generation stage, in response to the intermediate generated content of the i-th generation stage reaching the pause interaction condition, the video generation of the i-th generation stage is paused. Display the intermediate generated content, and display the pause evaluation question constructed based on the pause interaction conditions and the intermediate generated content; The continuing generation unit 19022 is specifically used for: In response to the feedback information regarding the pause evaluation issue, the intermediate generated content is accepted, and video generation in the i-th generation stage continues based on the intermediate generated content.

[0321] As one possible implementation, the display unit 19021 is specifically used for: Display the L stage results output by the i-th generation stage and the effect evaluation questions for the L stage results respectively, where L>1; The regeneration unit 19023 is specifically used for: The feedback information in response to the effect evaluation problem is to accept Q stage results out of the L stage results, generate video for the (i+1)th generation stage based on the Q stage results, and delete the other stage results from the L stage results except for the Q stage results, where 1 ≤ Q. <L。

[0322] As one possible implementation, the device 1900 further includes a multiple result generation triggering unit, used for: In response to the initial stage result of the i-th generation stage reaching a confidence condition relative to the content confidence of the video description text, the L-stage results are generated.

[0323] As one possible implementation, the device 1900 further includes a text updating unit for: In response to the feedback information of the problem to be evaluated, which is a content adjustment instruction for the generated content, the video description text is updated according to the content adjustment instruction. The updated video description text is applied to the video generation in the generation stages after the i-th generation stage in the N generation stages.

[0324] As one possible implementation, the display unit 19021 is specifically used for: The interactive page displays the generated content of the i-th generation stage when the interactive conditions are met, and the questions to be evaluated for the generated content. The device 1900 further includes a feedback information acquisition unit, used for: The feedback information for the question to be evaluated is obtained through the interactive page. The feedback information includes at least one of the following: content adjustment text or content adjustment annotations made on the generated content.

[0325] As one possible implementation, the device 1900 further includes a related stage regeneration unit for: In response to the feedback information of the problem to be evaluated, which is a content adjustment instruction for the generated content, the target video element involved in the content adjustment instruction is determined. In response to the fact that the target video element is related to the stage result of the j-th generation stage, the video generation of the j-th generation stage is restarted based on the content adjustment prompt, where 1 ≤ j. <i。

[0326] As one possible implementation, the device 1900 further includes a problem prioritization unit for: Generate candidate issues to be evaluated corresponding to the generated content; Based on the content elements involved in the candidate questions to be evaluated, determine the element confidence of the content elements relative to the video description text and the element type weight of the content elements; The display priority of the candidate questions to be evaluated is determined by the confidence level of the element, the weight of the element type, and the associated candidate questions to be evaluated. The R candidate problems with the highest display priority are selected as the problems to be evaluated.

[0327] As one possible implementation, the candidate problem to be evaluated is a closed-ended structured problem.

[0328] As one possible implementation, the device 1900 further includes a problem adjustment unit for: In response to the feedback quality of the obtained feedback information, the problem display parameters for the next display of the problem to be evaluated are adjusted. The feedback quality includes at least one of the interval between displaying the problem to be evaluated and obtaining the corresponding feedback information, or the richness of the content of the feedback information. The problem display parameters include at least one of the number of problems to be evaluated displayed, the display frequency, or the query granularity of the problems to be evaluated.

[0329] As one possible implementation, the generated content is displayed in the following manner: Obtain the original generated content of the i-th generation stage when the interaction condition is met; The generated content is previewed by reducing the rendering quality of the original generated content.

[0330] As one possible implementation, the device 1900 further includes a full-view display unit for: In response to the feedback information of the question to be evaluated, the generated content is accepted and the original generated content is displayed.

[0331] As one possible implementation, the device further includes a self-training unit for: Training samples are constructed based on the feedback information; The training samples are used to optimize the video generation model for generating the target video and the question generation model for generating the question to be evaluated.

[0332] As one possible implementation, the device 1900 further includes a backtracking unit for: In response to the stage backtracking operation, the historical generation stages for which video generation has been completed for the target video are displayed; Obtain a first content adjustment instruction for the first historical generation stage in the historical generation stage, and regenerate the video for the first historical generation stage and subsequent historical generation stages based on the first content adjustment instruction. The subsequent historical generation stage is a historical generation stage whose execution order is later than that of the first historical generation stage.

[0333] As one possible implementation, the device 1900 further includes a stage division unit, used for: The video generation process is divided into stages, resulting in M ​​generation stages involving the global static design of the video and P generation stages involving the local dynamic content of the video, where 1≤M, 1≤P, and M+P=N. The execution order of the N generation stages is determined based on the execution dependencies between the M generation stages and the P generation stages, wherein the execution order of the M generation stages is earlier than that of the P generation stages.

[0334] As one possible implementation, the N generation stages are based on the execution order and are, in order, the global planning stage, the keyframe generation stage, the motion planning stage, the fragment rendering stage, and the stitching optimization stage. The global planning stage is used to generate the character settings, scene settings, and storyboard settings for the target video; the keyframe generation stage is used to generate the keyframe sequence corresponding to the storyboard settings for the target video; the motion planning stage is used to generate motion planning information for the target video corresponding to the keyframe sequence and storyboard settings; the segment rendering stage is used to generate video segments corresponding to the storyboard settings for the target video; and the splicing optimization stage is used to obtain the target video by splicing the video segments.

[0335] This application also provides a computer device, including a terminal device or a server, in which the aforementioned video generation apparatus can be configured. The computer device will now be described in conjunction with the accompanying drawings.

[0336] If the computer device is a terminal device, please refer to Figure 20 As shown, this application provides a terminal device, taking a mobile phone as an example: Figure 20 The diagram shown is a block diagram of a portion of the structure of a mobile phone provided in an embodiment of this application. (Reference) Figure 20 The mobile phone includes components such as a radio frequency (RF) circuit 1410, a memory 1420, an input unit 1430, a display unit 1440, a sensor 1450, an audio circuit 1460, a Wi-Fi module 1470, a processor 1480, and a power supply 1490. Those skilled in the art will understand that... Figure 20 The mobile phone structure shown does not constitute a limitation on the mobile phone and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0337] The following is combined with Figure 20 A detailed introduction to each component of a mobile phone: The RF circuit 1410 can be used to receive and transmit signals during information transmission or calls. In particular, it receives downlink information from the base station and processes it with the processor 1480; in addition, it transmits uplink data to the base station.

[0338] The memory 1420 can be used to store software programs and modules. The processor 1480 executes various mobile phone functions and data processing by running the software programs and modules stored in the memory 1420. The memory 1420 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, applications required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory 1420 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0339] The input unit 1430 can be used to receive input numeric or character information, and to generate key signal inputs related to user settings and function control of the mobile phone. Specifically, the input unit 1430 may include a touch panel 1431 and other input devices 1432.

[0340] The display unit 1440 can be used to display information input by the user or information provided to the user, as well as various menus of the mobile phone. The display unit 1440 may include a display panel 1441.

[0341] The mobile phone may also include at least one sensor 1450, such as a light sensor, a motion sensor, and other sensors.

[0342] Audio circuitry 1460, speaker 1461, and microphone 1462 provide an audio interface between the user and the mobile phone.

[0343] WiFi is a short-range wireless transmission technology. Through the WiFi module 1470, mobile phones can help users send and receive emails, browse web pages, and access streaming media, providing users with wireless broadband internet access.

[0344] The processor 1480 is the control center of the mobile phone. It connects to various parts of the mobile phone through various interfaces and lines. It performs various functions of the mobile phone and processes data by running or executing software programs and / or modules stored in the memory 1420 and calling data stored in the memory 1420.

[0345] The phone also includes a power supply 1490 (such as a battery) that powers the various components.

[0346] In this embodiment, the processor 1480 included in the terminal device is also used to execute the steps in the methods of the various embodiments of this application.

[0347] If the computer device is a server, this application embodiment also provides a server; please refer to [link to relevant documentation]. Figure 21 As shown, Figure 21 This is a structural diagram of a server 1500 provided in an embodiment of this application. The server 1500 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 1522 (e.g., one or more processors) and memory 1532, and one or more storage media 1530 (e.g., one or more mass storage devices) for storing application programs 1542 or data 1544. The memory 1532 and storage media 1530 can be temporary or persistent storage. The program stored in the storage media 1530 may include one or more modules (not shown in the diagram), each module including a series of instruction operations on the server. Furthermore, the CPU 1522 may be configured to communicate with the storage media 1530 and execute the series of instruction operations in the storage media 1530 on the server 1500.

[0348] Server 1500 may also include one or more power supplies 1526, one or more wired or wireless network interfaces 1550, one or more input / output interfaces 1558, and / or one or more operating systems 1541, such as Windows Server. TM Mac OS X TM Unix TM Linux TM FreeBSD TM etc.

[0349] The steps performed by the server in the above embodiments can be based on Figure 21 The server structure shown.

[0350] In addition, this application embodiment also provides a storage medium for storing a computer program for executing the method provided in the above embodiment.

[0351] This application also provides a computer program product including a computer program, which, when run on a computer device, causes the computer device to perform the method provided in the above embodiments.

[0352] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium can be at least one of the following media: read-only memory (ROM), RAM, magnetic disk or optical disk, and other media that can store computer programs.

[0353] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0354] It should be noted that the various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for the device and system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiments. The device and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the solution in this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0355] The above description is merely one specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Moreover, based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods.

Claims

1. A video generation method, characterized in that, The method includes: Obtain the video description text used to describe the target video; Based on the video description text, video generation is performed sequentially in N stages for the target video, where N>1; The target video is obtained through video generation in the Nth generation stage; In the video generation of the N generation stages: When generating video in the i-th generation stage, in response to the achievement of the interaction condition, the generated content of the i-th generation stage at the time the interaction condition is met and the evaluation questions for the generated content are displayed, 1≤i <N; In response to the feedback information of the problem to be evaluated, the generated content is accepted and video generation in the i-th generation stage is continued based on the generated content, or video generation in the (i+1)-th generation stage is performed based on the generated content; In response to the feedback information of the problem to be evaluated, a content adjustment instruction is given for the generated content, and the video generation in the i-th generation stage is performed again based on the content adjustment instruction.

2. The method according to claim 1, characterized in that, The interaction condition is the completion of video generation in the i-th generation stage. Displaying the generated content of the i-th generation stage upon reaching the interaction condition and the evaluation questions for the generated content includes: Display the stage results output by the i-th generation stage and the effect evaluation question for the stage results; The feedback information in response to the problem to be evaluated is either accepting the generated content and continuing video generation in the i-th generation stage based on the generated content, or performing video generation in the (i+1)-th generation stage based on the generated content, including: The feedback information in response to the effect evaluation question is to accept the stage results and generate video for the (i+1)th generation stage based on the stage results.

3. The method according to claim 1, characterized in that, The interaction condition is a confidence condition for content confidence. In response to the achievement of the interaction condition, the generated content of the i-th generation stage at the time the interaction condition is met, and the questions to be evaluated regarding the generated content, include: During the video generation process of the i-th generation stage, in response to the fact that the content confidence of the intermediate generated content of the i-th generation stage relative to the video description text reaches the confidence condition, it is determined that the intermediate generated content has a content drift event, and the video generation of the i-th generation stage is paused. Display the intermediate generated content, and display the drift evaluation question constructed for the content drift event and the intermediate generated content.

4. The method according to claim 1, characterized in that, The interaction conditions are preset conditions for semantic changes in the content. In response to the achievement of the interaction conditions, the generated content of the i-th generation stage at the time the interaction conditions are met, and the questions to be evaluated regarding the generated content, include: During the video generation process of the i-th generation stage, in response to the semantic change of the intermediate generated content of the i-th generation stage reaching a preset condition, it is determined that a content switching event has occurred in the intermediate generated content, and the video generation of the i-th generation stage is paused. Display the intermediate generated content, and display the switching evaluation question constructed for the content switching event and the intermediate generated content; The feedback information in response to the problem to be evaluated is either accepting the generated content and continuing video generation in the i-th generation stage based on the generated content, or performing video generation in the (i+1)-th generation stage based on the generated content, including: In response to the feedback information of the switching evaluation problem, the intermediate generated content is accepted, and video generation in the i-th generation stage continues based on the intermediate generated content.

5. The method according to claim 1, characterized in that, The video description text also includes a pause interaction condition, the interaction condition including the pause interaction condition, and the response to reaching the interaction condition, displaying the generated content of the i-th generation stage at the time the interaction condition is reached and the evaluation questions for the generated content, including: During the video generation process of the i-th generation stage, in response to the intermediate generated content of the i-th generation stage reaching the pause interaction condition, the video generation of the i-th generation stage is paused. Display the intermediate generated content, and display the pause evaluation question constructed based on the pause interaction conditions and the intermediate generated content; The feedback information in response to the problem to be evaluated is either accepting the generated content and continuing video generation in the i-th generation stage based on the generated content, or performing video generation in the (i+1)-th generation stage based on the generated content, including: In response to the feedback information regarding the pause evaluation issue, the intermediate generated content is accepted, and video generation in the i-th generation stage continues based on the intermediate generated content.

6. The method according to claim 2, characterized in that, The process of displaying the stage results output by the i-th generation stage and evaluating the effectiveness of the stage results includes: Display the L stage results output by the i-th generation stage and the effect evaluation questions for the L stage results respectively, where L>1; The feedback information in response to the effect evaluation question is to accept the stage result and perform video generation for the (i+1)th generation stage based on the stage result, including: The feedback information in response to the effect evaluation problem is to accept Q stage results out of the L stage results, generate video for the (i+1)th generation stage based on the Q stage results, and delete the other stage results from the L stage results except for the Q stage results, where 1 ≤ Q. <L。 7. The method according to claim 6, characterized in that, The method further includes: In response to the initial stage result of the i-th generation stage reaching a confidence condition relative to the content confidence of the video description text, the L-stage results are generated.

8. The method according to any one of claims 1-7, characterized in that, The method further includes: In response to the feedback information of the problem to be evaluated, which is a content adjustment instruction for the generated content, the video description text is updated according to the content adjustment instruction. The updated video description text is applied to the video generation in the generation stages after the i-th generation stage in the N generation stages.

9. The method according to any one of claims 1-7, characterized in that, The display of the generated content in the i-th generation stage when the interaction condition is met, and the evaluation questions for the generated content, includes: The interactive page displays the generated content of the i-th generation stage when the interactive conditions are met, and the questions to be evaluated for the generated content. The method further includes: The feedback information for the question to be evaluated is obtained through the interactive page. The feedback information includes at least one of the following: content adjustment text or content adjustment annotations made on the generated content.

10. The method according to any one of claims 1-7, characterized in that, The method further includes: In response to the feedback information of the problem to be evaluated, which is a content adjustment instruction for the generated content, the target video element involved in the content adjustment instruction is determined. In response to the fact that the target video element is related to the stage result of the j-th generation stage, the video generation of the j-th generation stage is restarted based on the content adjustment prompt, where 1 ≤ j. <i。 11. The method according to claim 1, characterized in that, The method further includes: Generate candidate issues to be evaluated corresponding to the generated content; Based on the content elements involved in the candidate questions to be evaluated, determine the element confidence of the content elements relative to the video description text and the element type weight of the content elements; The display priority of the candidate questions to be evaluated is determined by the confidence level of the element, the weight of the element type, and the associated candidate questions to be evaluated. The R candidate problems with the highest display priority are selected as the problems to be evaluated.

12. The method according to claim 11, characterized in that, The candidate problems to be evaluated are closed-ended structured problems.

13. The method according to claim 1, characterized in that, The method further includes: In response to the feedback quality of the obtained feedback information, the problem display parameters for the next display of the problem to be evaluated are adjusted. The feedback quality includes at least one of the interval between displaying the problem to be evaluated and obtaining the corresponding feedback information, or the richness of the content of the feedback information. The problem display parameters include at least one of the number of problems to be evaluated displayed, the display frequency, or the query granularity of the problems to be evaluated.

14. The method according to any one of claims 1-7, characterized in that, The generated content is displayed in the following manner: Obtain the original generated content of the i-th generation stage when the interaction condition is met; The generated content is previewed by reducing the rendering quality of the original generated content.

15. The method according to claim 14, characterized in that, The method further includes: In response to the feedback information of the question to be evaluated, the generated content is accepted and the original generated content is displayed.

16. The method according to any one of claims 1-7, characterized in that, The method further includes: Training samples are constructed based on the feedback information; The training samples are used to optimize the video generation model for generating the target video and the question generation model for generating the question to be evaluated.

17. The method according to any one of claims 1-7, characterized in that, The method further includes: In response to the stage backtracking operation, the historical generation stages for which video generation has been completed for the target video are displayed; Obtain a first content adjustment instruction for the first historical generation stage in the historical generation stage, and regenerate the video for the first historical generation stage and subsequent historical generation stages based on the first content adjustment instruction. The subsequent historical generation stage is a historical generation stage whose execution order is later than that of the first historical generation stage.

18. The method according to any one of claims 1-7, characterized in that, The method further includes: The video generation process is divided into stages, resulting in M ​​generation stages involving the global static design of the video and P generation stages involving the local dynamic content of the video, where 1≤M, 1≤P, and M+P=N. The execution order of the N generation stages is determined based on the execution dependencies between the M generation stages and the P generation stages, wherein the execution order of the M generation stages is earlier than that of the P generation stages.

19. The method according to claim 18, characterized in that, The N generation stages are based on the execution order and are, in order, the global planning stage, the keyframe generation stage, the motion planning stage, the fragment rendering stage, and the stitching optimization stage. The global planning stage is used to generate the character settings, scene settings, and storyboard settings for the target video; the keyframe generation stage is used to generate the keyframe sequence corresponding to the storyboard settings for the target video; the motion planning stage is used to generate motion planning information for the keyframe sequence and storyboard settings for the target video; and the segment rendering stage is used to generate video segments corresponding to the storyboard settings for the target video. The splicing optimization stage is used to obtain the target video by splicing the video segments.

20. A video generation apparatus, characterized in that, The device includes an acquisition unit and a generation unit; The acquisition unit is used to acquire video description text used to describe the target video; The generation unit is configured to sequentially generate video for the target video in N generation stages based on the video description text, where N>1; The generation unit is further configured to obtain the target video through video generation in the Nth generation stage; In the video generation process involving the N generation stages, the generation unit includes a display unit, a continued generation unit, and a regeneration unit: The display unit is configured to, when video generation is performed in the i-th generation stage, display the generated content of the i-th generation stage at the time the interaction condition is met, and the evaluation question for the generated content, 1≤i, in response to the achievement of the interaction condition. <N; The continuing generation unit is configured to respond to the feedback information of the problem to be evaluated as accepting the generated content, and continue video generation in the i-th generation stage based on the generated content, or perform video generation in the (i+1)-th generation stage based on the generated content; The regeneration unit is configured to respond to the feedback information of the problem to be evaluated as a content adjustment instruction for the generated content, and regenerate the video in the i-th generation stage based on the content adjustment instruction.

21. A computer device, characterized in that, The computer device includes a processor and memory: The memory is used to store computer programs; The processor is configured to perform the method according to any one of claims 1-19 according to the computer program.

22. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store a computer program that, when executed by a computer device, performs the method described in any one of claims 1-19.

23. A computer program product comprising a computer program, which, when run on a computer device, causes the computer device to perform the method of any one of claims 1-19.

Images