An interactive agent feedback video generation method

Abstract

The application relates to the technical field of video generation, in particular to a video generation method of interactive Agent feedback, which comprises the following steps: controlling a first intelligent agent to generate a shot script according to user input content, and detecting based on a first gating mechanism; controlling a second intelligent agent to generate a plurality of key frames according to the shot script, and detecting based on a second gating mechanism; inputting a third intelligent agent according to the key frames to generate and render interframe video, and detecting based on a third gating mechanism; in view of the problem that the text-to-video system in the prior art has a long rework time in the full-amount generation process, an evaluation intelligent agent is introduced in combination with a plurality of gating mechanisms, the corresponding gating mechanisms are respectively inserted according to the shot script, the key frames and the interframe video, the corresponding gating stages are subjected to rule discrimination through the evaluation intelligent agent, and it is judged whether the preset rules or gating scores are met, the phased checking and the correction process are realized, and therefore the interception of the directional error is moved forward to the low-computing-power stage.

Description

Technical Field

[0001] This invention relates to the field of video generation technology, and more specifically to a video generation method with interactive agent feedback. Background Technology

[0002] In recent years, with the rapid development of diffusion models and Transformer architectures, text-to-video (T2V) generation technology has made groundbreaking progress. Current mainstream T2V systems typically employ a cascaded architecture. First, a large language model expands the user's text requirements into a detailed video description or storyboard. Then, a text encoder injects semantic information into a spatiotemporal diffusion model, gradually denoising and generating a video frame sequence in the latent space. Finally, a video decoder maps the sequence to pixel space to output the final video. Some systems also introduce dynamic frame interpolation and super-resolution modules to improve temporal smoothness and spatial clarity.

[0003] For example, patent application CN202610602641.7 discloses an iterative self-optimizing text-based video generation method and system based on knowledge enhancement, relating to the field of video generation technology. The method includes: acquiring the user's original text input; matching physical rule sets from a static physical knowledge base and recalling historical constraint sets from a dynamic constraint memory base; performing physical rule constraints and memory fusion to generate knowledge-enhanced video generation prompts; inputting the generated prompts into a pre-trained diffusion-based T2V model to generate the target video; performing static appearance verification and dynamic physical verification on the target video, and calculating semantic consistency scores and physical common sense scores; based on the verification results and scores, determining whether the iteration termination condition is met; if so, outputting the final video and optimized prompts; forming a complete closed loop of "knowledge retrieval - planning generation - verification feedback - memory accumulation," significantly improving the generalization ability of out-of-distribution scenarios and achieving continuous iterative improvement in the system's generation quality.

[0004] For example, patent application CN202511947913.9 discloses a text-based video model acceleration system and method based on dual sparsity. The system includes an inter-frame sequence compression module and a sparse full attention module. The inter-frame sequence compression module is configured to: acquire a video frame sequence to be processed; determine a reference frame and a frame to be compressed based on the video frame sequence; calculate the target cosine similarity between each token in the frame to be compressed and the corresponding token or region in the reference frame; and merge the tokens in the reference frame and the frame to be compressed based on the target cosine similarity. The sparse full attention module is configured to: determine an attention matrix based on the merged video frame sequence; generate a binary block mask based on the attention matrix; and perform attention calculation on the attention matrix based on the binary block mask to obtain the calculation result. This addresses the problems of slow inference speed, high computational resource consumption, and the inability of current T2V model acceleration schemes to balance efficiency and quality.

[0005] However, in actual implementation, the inventors found that this type of technical solution uses text prompts as the main control method and usually follows a linear processing path during the generation process. When video clips or keyframes do not meet expectations, they must be completely regenerated, which leads to a long production time and the need for multiple reworks. Summary of the Invention

[0006] To address the aforementioned problems in existing technologies, a video generation method based on interactive agent feedback is provided.

[0007] The specific technical solution is as follows: A method for generating video feedback from an interactive agent includes: Step S1: Control the first intelligent agent to generate a storyboard script based on the user input, generate a gate score for the storyboard script based on the first gate control mechanism, and determine whether the gate score is greater than the gate control threshold. If so, proceed to step S2; If not, generate the parameter correction vector and then return to step S1 to regenerate the storyboard; Step S2: Control the second intelligent agent to generate multiple key frames according to the storyboard script, and detect each key frame separately based on the second gating mechanism, and determine whether the first detection result conforms to the first preset rule; If so, proceed to step S3; If not, generate a structured correction instruction and then return to step S2 to regenerate the keyframe; The first detection result includes color difference detection result and face consistency detection result; Step S3: The third agent is input based on the key frame to generate and render inter-frame video. Then, each inter-frame video is detected based on the third gating mechanism, and it is determined whether the second detection result conforms to the second preset rule. If so, output the final product; If not, generate structured correction instructions and then return to step S3 to render the failed inter-frame video; In step S3, only the inter-frame videos that failed are regenerated; the second detection result includes inter-frame consistency detection result, cross-shot character consistency detection result, and motion smoothing detection result; The first gating mechanism, the second gating mechanism, and the third gating mechanism are implemented based on a review agent; The parameter correction vector is generated based on at least one of multimodal inverse deconstruction, stage-gated state machine, and natural language feedback.

[0008] On the other hand, step S1 includes: Step S11: Obtain the user input content and input it into the first intelligent agent to generate the storyboard script; Step S12: Combine the user input content with the storyboard input review agent to generate the gate score according to the dimensions included in the first gate mechanism, and determine whether the gate score is greater than the gate threshold; If so, proceed to step S2; If not, proceed to step S13; Step S13: Obtain the user-guided content to generate a parameter correction vector, and then return to step S11 to input the user input content and the parameter correction vector together into the first intelligent agent.

[0009] On the other hand, step S13 includes: Step S131: Feed back the gate control score to the user to obtain the user guidance content; Step S132: Input the user guidance content into the large model to obtain structured correction instructions; Step S133: Based on the mapping function, map the structured instructions and the user input content together into the parameter correction vector.

[0010] On the other hand, the dimensions of the parameter correction vector include: brightness adjustment value, hue shift value, motion speed correction parameter, style shift parameter, and facial reference parameter.

[0011] On the other hand, before performing step S1, the procedure also includes: Step S01: Obtain the reference video, extract video description information from the reference video and generate reference style information; The video description information includes at least one of the following: transition type parameters, 3D LUT parameters, camera motion trajectory descriptors, and structured story information; In step S11, the reference style information and the user input content are input together into the first intelligent agent.

[0012] On the other hand, in step S13, the parameter correction vector is generated based on the user guidance content and the reference style information.

[0013] On the other hand, step S2 includes: Step S21: Extract keyframe generation nodes from the storyboard script, extract descriptive text based on the keyframe generation nodes, and construct keyframe prompts; Step S22: Input the keyframe prompts into the second agent to obtain multiple keyframes respectively; Step S23: The review agent detects the key frames based on the second gating mechanism and determines whether the first detection results all conform to the preset rules. If so, proceed to step S3; If not, return to step S2 to regenerate the keyframe.

[0014] On the other hand, in step S21, the keyframe cue words are also constructed by extracting tone information and style embedding information based on the reference style information.

[0015] On the other hand, step S3 includes: Step S31: Use the keyframe as a reference and the lens motion trajectory descriptor as motion guide to generate a video clip; Step S32: Splice the video segments according to the transition type parameters to obtain a spliced ​​video; Step S33: Apply the 3D LUT parameters to the spliced ​​video, then detect the inter-frame video corresponding to each video segment based on the third gating mechanism, and determine whether the second detection result conforms to the second preset rule; If so, the stitched video with the 3D LUT parameters applied is output as the final product; If not, return to step S31 to re-render the failed inter-frame video.

[0016] A video generation system for implementing a video generation method; The video generation system includes: The multimodal reverse deconstruction module is used to extract transition type parameters, 3D LUT parameters, camera motion trajectory descriptors, and structured story information from the reference video provided by the user, and generate reference style information. The gated pipeline scheduling module is used to schedule the first, second, and third agents in the order of the text script stage, keyframe image stage, and video rendering stage, and control the stage advancement or stage rollback based on the gating results of each stage. The first agent generates a storyboard script during the text script stage, the second agent generates keyframes during the keyframe image stage, and the third agent generates inter-frame video during the video rendering stage. The review agent module detects the storyboard, the keyframes, and the inter-frame video based on the review agent and generates the gating result. Furthermore, when the gating result is "not passed", the review agent gating will parse the user's natural language feedback into a structured correction instruction, and then generate a parameter correction vector through a mapping function; The local regeneration module is used to regenerate or re-render only the target keyframes or target video segments that fail the gating detection, based on the parameter correction vector. The video output module is used to splice the video segments according to the transition type parameters after the video segments pass the gating detection, and to output the finished product by applying the 3D LUT parameters.

[0017] The above technical solution has the following advantages or beneficial effects: To address the issue of existing text-to-video systems lacking error correction and adjustment mechanisms during the full generation process, resulting in long rework times, this solution introduces a review agent combined with multiple gating mechanisms. Corresponding gating mechanisms are inserted for storyboards, keyframes, and inter-frame videos. The review agent performs rule-based judgment on the corresponding gating stages and determines whether they meet preset rules or gating scores, thus realizing a phased verification and correction process. This allows the interception of directional errors to be moved forward to a low-computing-power stage. Attached Figure Description

[0018] Embodiments of the invention will be described more fully with reference to the accompanying drawings. However, the drawings are for illustration and explanation only and do not constitute a limitation on the scope of the invention.

[0019] Figure 1 This is an overall schematic diagram of an embodiment of the present invention; Figure 2 This is a schematic diagram of step S01 in an embodiment of the present invention; Figure 3 This is a schematic diagram of step S1 in an embodiment of the present invention; Figure 4 This is a schematic diagram of step S13 in an embodiment of the present invention; Figure 5 This is a schematic diagram of step S2 in an embodiment of the present invention; Figure 6 This is a schematic diagram of step S3 in an embodiment of the present invention; Figure 7 This is a schematic diagram of the system in an embodiment of the present invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0022] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, but this is not intended to limit the scope of the invention.

[0023] This invention includes: A video generation method based on interactive agent feedback, such as Figure 1 As shown, it includes: Step S1: Control the first intelligent agent to generate a storyboard script based on the user input, generate a gating score for the storyboard script based on the first gating mechanism, and determine whether the gating score is greater than the gating threshold. If so, proceed to step S2; If not, generate the parameter correction vector and then return to step S1 to regenerate the storyboard; Step S2: Control the second intelligent agent to generate multiple keyframes according to the storyboard script, and detect each keyframe separately based on the second gating mechanism, and determine whether the first detection result conforms to the first preset rule; If so, proceed to step S3; If not, generate structured correction instructions and then return to step S2 to regenerate the keyframe; The first detection result includes color difference detection result and face consistency detection result; Step S3: Based on the key frame input, the third agent generates and renders the inter-frame video. Then, based on the third gating mechanism, the inter-frame video is detected, and it is determined whether the second detection result conforms to the second preset rule. If so, output the final product; If not, generate structured correction instructions and then return to step S3 to render the failed inter-frame video; In step S3, only the inter-frame videos that failed are regenerated; The second detection result includes inter-frame consistency detection result, cross-shot character consistency detection result, and motion smoothness detection result; The first, second, and third gating mechanisms are implemented based on the review agent.

[0024] Specifically, addressing the issue that existing text-to-video systems lack error correction and adjustment mechanisms during the full generation process, resulting in long rework times, this solution introduces a review agent combined with multiple gating mechanisms. Corresponding gating mechanisms are inserted for storyboards, keyframes, and inter-frame videos. The review agent performs rule-based judgment on the corresponding gating stages and determines whether they meet preset rules or gating scores, thus realizing a phased verification and correction process. This allows the interception of directional errors to be moved forward to a low-computing-power stage.

[0025] In practical implementation, the above technical solution is mainly configured as a software system in a computer system to receive user instructions and generate corresponding video clips. The generation process of this software system is achieved by a first intelligent agent, a second intelligent agent, a third intelligent agent, and a review intelligent agent in conjunction with user input.

[0026] The system comprises three intelligent agents: the first agent, generated based on a large language model, capable of generating corresponding storyboards based on specific prompts; the second agent, a diffusion model based on a cross-attention mechanism, used to generate single-frame static images as keyframes from specific text segments in the storyboard corresponding to keyframes; and the third agent, implemented based on a temporal attention mechanism layer, capable of automatically generating video segments within a given interval based on two adjacent keyframes combined with the storyboard.

[0027] Correspondingly, the review model is a large language model with multimodal input. It uses a pre-configured gating mechanism to determine whether the output of the agent in each stage meets the preset score or preset rules, and provides the corrected prompts and parameters to the generation modules of each stage to form a correction closed loop. When there is no score improvement for two consecutive rounds, it actively suggests that the user accept the current result or change the reference style to prevent infinite loop.

[0028] During the evaluation process, the first gating mechanism includes a pre-set multi-dimensional scoring mechanism, encompassing four dimensions: plot coherence, pacing matching, shot richness, and tone consistency. The reviewing agent uses this first gating mechanism to score each dimension, then calculates a weighted sum to arrive at a gating score, determining whether the score exceeds a gating threshold, such as seven out of ten. The next stage of the generation process is then chosen, or the storyboard script is regenerated.

[0029] As an optional implementation, the multidimensional scoring mechanism can also allow users to input their scoring results.

[0030] After the second agent generates the corresponding keyframes, the review agent performs detection on the keyframes based on the second gating mechanism. This includes color difference detection on each keyframe based on the CIEDE2000 color difference formula, and cross-frame role consistency detection on faces appearing in each keyframe based on the cosine similarity of ArcFace face feature vectors. This yields the first detection result, including color difference detection results and face consistency detection results. The review agent then selectively generates keyframes that do not meet the requirements according to the gating mechanism until all keyframes meet the first preset rule.

[0031] As an optional implementation, the output of the review agent in the second gating mechanism can be fed back to the user as a prompt, allowing the user to control the actual keyframes that do not conform to the first detection rule and regenerate them.

[0032] After the third agent generates the corresponding inter-frame video based on each keyframe, the review agent performs inter-frame video detection based on the third gating mechanism, including inter-frame SSIM consistency detection to obtain inter-frame consistency detection results, ArcFace cross-camera character consistency detection to obtain cross-camera character consistency detection results, and motion trajectory smoothness detection to obtain motion smoothness detection results.

[0033] The generation process involves regenerating video segments that do not meet the second detection rule individually, and then outputting the spliced ​​final video after all segments pass the test.

[0034] The aforementioned gating mechanism is controlled by a computing power gating state machine during the generation process. A three-stage state machine is defined: S_text→S_image→S_video. The stage transition must meet the corresponding quality threshold conditions. If the conditions are not met, the process will fall back to the current stage for correction. The maximum number of iterations is K_max. If the number of iterations is exceeded, manual intervention will be required. The system records each gating interception event and accumulates the statistics, thereby saving computing power during generation.

[0035] In one embodiment, such as Figure 2 As shown, the procedure before performing step S1 also includes: Step S01: Obtain the reference video, extract video description information from the reference video and generate reference style information; The video description information includes at least one of the following: transition type parameters, 3D LUT parameters, camera motion trajectory descriptors, and structured story information.

[0036] In step S1, the reference style information and the user input are input into the first intelligent agent.

[0037] Specifically, in order to achieve better control over video style and uniformity of the final product, this embodiment also pre-acquires the reference video provided by the user and extracts the video description information to construct reference style information. In the subsequent generation process, the reference style information and the user input content are jointly input into the first intelligent agent to control the generation of content.

[0038] Specifically, the video description information includes at least one of the following: transition type parameters, 3D LUT parameters, camera motion trajectory descriptors, and structured story information.

[0039] Specifically, for the transition type parameter, candidate transition points are first detected using the frame difference energy function in the reference video, and the transition type is then classified by a lightweight CNN network. After extracting the candidate transition points, eight image frames before and after each candidate transition point are taken to form a transition segment, which is then fed into the lightweight CNN network for classification.

[0040] For the 3D LUT parameters, the tone histograms of uniformly sampled frames for each shot segment are statistically analyzed in the Lab color space. The center point of the histogram bin is used as the control point for thin plate spline interpolation to fit and generate 33×33×33 3D LUT parameters.

[0041] For the lens motion trajectory descriptor, the dense optical flow field of adjacent frames is calculated using the RAFT optical flow algorithm, and a 6-parameter affine motion model is fitted to obtain the lens motion trajectory descriptor.

[0042] Furthermore, for structured story information, scene descriptions are generated shot-by-shot using a multimodal large model, and then summarized by a large language model into structured storylines and rhythm parameters as structured story information. These features are aggregated into a JSON file as video description information.

[0043] In the subsequent generation process, structured story information can guide the generation of storyboards, while transition type parameters, 3DLUT parameters, and camera motion trajectory descriptors can be used to guide the third agent in generating corresponding video clips, camera movements, and color styles within the video clips, thereby enabling control over the video style.

[0044] In one embodiment, such as Figure 3 As shown, step S1 includes: Step S11: Obtain user input and input it into the first intelligent agent to generate a storyboard script; Step S12: Input the user input content and the storyboard script into the review agent to generate a gate score based on the dimensions included in the first gate mechanism, and determine whether the gate score is greater than the gate threshold. If so, proceed to step S2; If not, proceed to step S13; Step S13: Obtain the user-guided content and generate a parameter correction vector, then return to step S11 to input the user input content and the parameter correction vector together into the first agent.

[0045] Specifically, to achieve better control over the generated content, this embodiment first acquires user input and inputs it into a first intelligent agent to generate a storyboard. The first intelligent agent is generated based on a large language model, capable of generating a corresponding storyboard based on specific prompts. The storyboard includes specified scenes, characters, dialogues, and keyframes.

[0046] Subsequently, the user input and the storyboard script are input into a review AI agent to generate a gating score based on the dimensions included in the first gating mechanism. These dimensions include plot coherence, pacing matching, shot richness, and tone consistency. The review AI agent scores each dimension using the first gating mechanism, then calculates the gating score through weighted summation. It then determines whether the gating score exceeds the gating threshold, and subsequently chooses to proceed to the next stage of generation or revise the storyboard script.

[0047] If the storyboard script needs to be revised, the parameter revision vector generated from the user-guided content is obtained and input together with the user input into the first intelligent agent for feedback.

[0048] When performing step S11, if there is pre-extracted reference style information, then in step S11, the structured story information and the user input content are input into the first intelligent agent.

[0049] In one embodiment, such as Figure 4 As shown, step S13 includes: Step S131: Provide the gate control score to the user to obtain user guidance content; Step S132: Input the user guidance content into the large model to obtain structured correction instructions; Step S133: Based on the mapping function, the structured instructions and user input content are jointly mapped into a parameter correction vector.

[0050] Specifically, in order to provide feedback and correction on the storyboard script, this embodiment first provides the user with a gating score to obtain user guidance content, including sub-scores for four dimensions: plot coherence, rhythm matching, shot richness, and tone fit, to alert the user to relevant issues. Then, the user inputs the corresponding user guidance content themselves.

[0051] Then, the review agent inputs the user's natural language feedback into the large language model and outputs structured correction instructions, including fields such as the target scene number and an adjustment array. The adjustment array includes dimension names, adjustment directions, and adjustment intensities. Subsequently, based on a mapping function, the structured instructions and the prompts corresponding to the user's input are jointly mapped into a parameter correction vector. During the mapping process, the corresponding parameter correction vector is formed by superimposing the vector dimensions, and the dimensions to be corrected do not interfere with each other.

[0052] As an optional implementation, when reference style information is configured, the parameter correction vector is also generated jointly based on the user guidance content and the reference style information when generating the parameter correction vector.

[0053] In one embodiment, such as Figure 5 As shown, step S2 includes: Step S21: Extract keyframe generation nodes from the storyboard script, extract descriptive text based on the keyframe generation nodes, and construct keyframe cue words; Step S22: Input the keyframe prompts into the second agent to obtain multiple keyframes; Step S23: The review agent detects the key frames based on the second gating mechanism and determines whether the first detection results all conform to the first preset rules. If so, proceed to step S3; If not, return to step S21 to regenerate the keyframe.

[0054] Specifically, to achieve a better keyframe generation process, in this embodiment, keyframe generation nodes are first extracted from the storyboard script. Based on these nodes, descriptive text paragraphs are extracted and combined with pre-configured templates to construct keyframe prompts. The keyframe prompts are then input into a second agent to obtain multiple keyframes.

[0055] The second agent is a diffusion model based on a cross-attention mechanism, used to generate a single static image as a keyframe based on specific text paragraphs in the storyboard that correspond to keyframes.

[0056] Then, the review agent performs detection on the key frames based on the second gating mechanism, including color difference detection on each key frame based on the CIEDE2000 color difference formula, and cross-frame role consistency detection on the faces appearing in each key frame based on the cosine similarity of the ArcFace face feature vector, thereby obtaining the first detection result, including the color difference detection result and the face consistency detection result. Then, according to the gating mechanism, key frames that do not meet the requirements are selectively generated until all key frames meet the first preset rule.

[0057] In one embodiment, in step S21, tone information and style embedding information are extracted based on reference style information to jointly construct keyframe cue words.

[0058] Specifically, in the embodiment where a reference video is configured, when generating keyframe prompts, tone information and style embedding information are extracted from the reference style information to construct the prompts. The tone information is generated by reading the tone curves corresponding to the 3D LUT parameters and generating textual descriptive information. The style embedding information is obtained by vectorizing the reference style information, including transition type parameters, 3D LUT parameters, and camera motion trajectory descriptors, and then inputting it into a vector machine for classification. Through these processing steps, the numerical descriptive information is transformed into textual prompts, achieving better control over the style.

[0059] In one embodiment, such as Figure 6 As shown, step S3 includes: Step S31: Using keyframes as a reference and camera motion trajectory descriptors as motion guidance input, the third agent generates video clips; Step S32: Splice the video segments according to the transition type parameters to obtain a spliced ​​video; Step S33: Apply 3D LUT parameters to the spliced ​​video, then detect the inter-frame video corresponding to each video segment based on the third gating mechanism, and determine whether the second detection result conforms to the second preset rule; If so, output the stitched video with 3D LUT parameters as the final product; If not, return to step S31 to re-render the failed inter-frame video.

[0060] Specifically, to achieve better generation results, in this embodiment, keyframes are first used as a reference, and camera motion trajectory descriptors are used as motion guidance inputs to a third agent to generate video segments. The third agent is implemented based on a temporal attention mechanism layer and can automatically generate video segments within a range based on two adjacent keyframes combined with the storyboard.

[0061] Then, in the third intelligent agent, the video segments are spliced ​​together by adding transition type parameters to obtain a spliced ​​video, forming a preliminary full video, and the spliced ​​video is then subjected to 3D LUT parameters to complete tone control.

[0062] To facilitate local rendering corrections, after loading the 3D LUT parameters, corresponding annotations were added to the timeline of the stitched video and video segments. This allowed for the reverse extraction of corresponding inter-frame video for detection through a third gating mechanism. This included inter-frame consistency detection using SSIM, ArcFace cross-camera character consistency detection, and motion smoothness detection.

[0063] Finally, output the stitched video that meets the requirements as the final product or perform partial re-rendering.

[0064] In one embodiment, to achieve a faster local readjustment method, the correction process is also implemented by constructing specific structured instructions combined with another set of parameter correction vector dimensions.

[0065] The review agent inputs the user's natural language feedback into a large language model and outputs structured correction instructions, including fields such as the target scene number and an adjustment array. The adjustment array includes dimension names, adjustment directions, and adjustment intensities. Subsequently, based on a mapping function, the structured instructions and corresponding prompts from the user input are mapped together into a parameter correction vector. During the mapping process, the corresponding parameter correction vector is formed by superimposing the vector dimensions, ensuring that the dimensions to be corrected do not interfere with each other. The dimensions of the parameter correction vector include: brightness adjustment value, hue shift value, motion speed correction parameter, style shift parameter, and facial reference parameter. This allows for targeted adjustments to specific parts of the video.

[0066] The adjustment process is shown in Table 1: Table 1

[0067] In actual judgment, the scoring thresholds are shown in Table 2:

[0068] A storage medium includes computer instructions that, when executed by a computer device, perform the video generation method described above.

[0069] A video generation system for implementing a video generation method; like Figure 7 As shown, the video generation system includes: The multimodal reverse deconstruction module 1 is used to extract transition type parameters, 3D LUT parameters, camera motion trajectory descriptors, and structured story information from the reference video provided by the user, and generate reference style information. The gated pipeline scheduling module 2 is used to schedule the first agent, the second agent, and the third agent in the order of the text script stage, the key frame image stage, and the video rendering stage, and to control the stage advancement or stage rollback based on the gating results of each stage. The first agent generates the storyboard script during the text script stage, the second agent generates keyframes during the keyframe image stage, and the third agent generates inter-frame video during the video rendering stage. Review agent module 3: Review agent module 3 detects storyboards, keyframes and inter-frame videos based on the review agent and generates gating results. Furthermore, when the gating result is "not passed", the review agent module 3 parses the user's natural language feedback into structured correction instructions, and then generates parameter correction vectors through a mapping function; Local regeneration module 4 is used to regenerate or re-render only the target keyframes or target video segments that have not passed the gating detection, based on the parameter correction vector. The video output module 5 is used to splice video segments according to transition type parameters after the video segments pass through the gating detection, and then apply 3D LUT parameters to output the final product.

[0070] Those skilled in the art will understand that various aspects, or possible implementations of various aspects, of the present invention can be embodied as systems, methods, or computer program products. Therefore, various aspects, or possible implementations of various aspects, of the present invention can take the form of entirely hardware embodiments, entirely software embodiments (including firmware, resident software, etc.), or embodiments combining software and hardware aspects, all collectively referred to herein as "circuit," "module," or "system." Furthermore, various aspects, or possible implementations of various aspects, of the present invention can take the form of computer program products, which are computer instructions stored in memory.

[0071] The memory can be a computer-readable signal medium or a computer-readable storage medium. Computer-readable storage media include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or apparatuses, or any suitable combination thereof, such as random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, and portable read-only memory (CD-ROM).

[0072] A processor in a computer reads computer instructions stored in memory, enabling the processor to execute the functional actions specified in each step or combination of steps in a flowchart; and to generate means for implementing the functional actions specified in each block or combination of blocks in a flowchart.

[0073] It should be understood that a processor in a computer can be understood as one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers (MCUs), microprocessors, or other electronic components used to execute the aforementioned computer instructions.

[0074] Computer instructions may be executed entirely on the user's local computer, partially on the user's local computer, as a separate software package, partially on the user's local computer and partially on a remote computer, or entirely on a remote computer or server. It should also be noted that in some alternative implementations, the functions indicated by the steps in the flowchart or the blocks in the block diagram may not occur in the order shown in the diagram. For example, depending on the functions involved, two consecutive steps or blocks may actually be executed approximately simultaneously, or these blocks may sometimes be executed in reverse order.

[0075] Of course, in practical applications, the various components of a computer system are coupled together through a bus system. The bus system is used to enable communication and connection between these components. In addition to the data bus, the bus system also includes a power bus, a control bus, and a status signal bus.

[0076] The above are merely preferred embodiments of the present invention and are not intended to limit the implementation methods and protection scope of the present invention. Those skilled in the art should recognize that any equivalent substitutions and obvious changes made based on the description and illustrations of the present invention should be included within the protection scope of the present invention.

Claims

1. A video generation method with interactive agent feedback, characterized in that, include: Step S1: Control the first intelligent agent to generate a storyboard script based on the user input, generate a gate score for the storyboard script based on the first gate control mechanism, and determine whether the gate score is greater than the gate control threshold. If so, proceed to step S2; If not, generate the parameter correction vector and then return to step S1 to regenerate the storyboard; Step S2: Control the second intelligent agent to generate multiple key frames according to the storyboard script, and detect each key frame separately based on the second gating mechanism, and determine whether the first detection result conforms to the first preset rule; If so, proceed to step S3; If not, generate a structured correction instruction and then return to step S2 to regenerate the keyframe; The first detection result includes color difference detection result and face consistency detection result; Step S3: The third agent is input based on the key frame to generate and render inter-frame video. Then, each inter-frame video is detected based on the third gating mechanism, and it is determined whether the second detection result conforms to the second preset rule. If so, output the final product; If not, generate structured correction instructions and then return to step S3 to render the failed inter-frame video; In step S3, only the inter-frame videos that failed are regenerated; The second detection result includes inter-frame consistency detection result, cross-shot character consistency detection result, and motion smoothing detection result; The first gating mechanism, the second gating mechanism, and the third gating mechanism are implemented based on a review agent; The parameter correction vector is generated based on at least one of multimodal inverse deconstruction, stage-gated state machine, and natural language feedback.

2. The video generation method according to claim 1, characterized in that, Step S1 includes: Step S11: Obtain the user input content and input it into the first intelligent agent to generate the storyboard script; Step S12: Combine the user input content with the storyboard input review agent to generate the gate score according to the dimensions included in the first gate mechanism, and determine whether the gate score is greater than the gate threshold; If so, proceed to step S2; If not, proceed to step S13; Step S13: Obtain the user-guided content to generate a parameter correction vector, and then return to step S11 to input the user input content and the parameter correction vector together into the first intelligent agent.

3. The video generation method according to claim 2, characterized in that, Step S13 includes: Step S131: Feed back the gate control score to the user to obtain the user guidance content; Step S132: Input the user guidance content into the large model to obtain structured correction instructions; Step S133: Based on the mapping function, map the structured instructions and the user input content together into the parameter correction vector.

4. The video generation method according to claim 3, characterized in that, The structured correction instruction includes fields such as target storyboard number and adjustment array; The adjustment array includes dimension name, adjustment direction, and adjustment intensity.

5. The video generation method according to claim 3, characterized in that, The procedure before performing step S1 also includes: Step S01: Obtain the reference video, extract video description information from the reference video and generate reference style information; The video description information includes at least one of the following: transition type parameters, 3D LUT parameters, camera motion trajectory descriptors, and structured story information; In step S11, the reference style information and the user input content are input together into the first intelligent agent.

6. The video generation method according to claim 5, characterized in that, In step S13, the parameter correction vector is generated based on the user guidance content and the reference style information.

7. The video generation method according to claim 5, characterized in that, Step S2 includes: Step S21: Extract keyframe generation nodes from the storyboard script, extract descriptive text based on the keyframe generation nodes, and construct keyframe prompts; Step S22: Input the keyframe prompts into the second agent to obtain multiple keyframes respectively; Step S23: The review agent detects the keyframes based on the second gating mechanism and determines whether the first detection results all conform to the first preset rule. If so, proceed to step S3; If not, return to step S21 to regenerate the keyframe.

8. The video generation method according to claim 7, characterized in that, In step S21, the keyframe cue words are also constructed by extracting tone information and style embedding information based on the reference style information.

9. The video generation method according to claim 5, characterized in that, Step S3 includes: Step S31: Using the keyframe as a reference and the camera motion trajectory descriptor as motion guidance input, the third agent generates a video clip; Step S32: Splice the video segments according to the transition type parameters to obtain a spliced ​​video; Step S33: Apply the 3D LUT parameters to the spliced ​​video, then detect the inter-frame video corresponding to each video segment based on the third gating mechanism, and determine whether the second detection result conforms to the second preset rule; If so, the stitched video with the 3D LUT parameters applied is output as the final product; If not, return to step S31 to re-render the failed inter-frame video.

10. A video generation system, characterized in that, Used to implement the video generation method as described in any one of claims 1-9; The video generation system includes: The multimodal reverse deconstruction module is used to extract transition type parameters, 3D LUT parameters, camera motion trajectory descriptors, and structured story information from the reference video provided by the user, and generate reference style information. The gated pipeline scheduling module is used to schedule the first, second, and third agents in the order of the text script stage, keyframe image stage, and video rendering stage, and control the stage advancement or stage rollback based on the gating results of each stage. The first agent generates a storyboard script during the text script stage, the second agent generates keyframes during the keyframe image stage, and the third agent generates inter-frame video during the video rendering stage. The review agent module detects the storyboard, the keyframes, and the inter-frame video based on the review agent and generates the gating result. Furthermore, when the gating result is "not passed", the review agent module parses the user's natural language feedback into a structured correction instruction, and then generates a parameter correction vector through a mapping function; The local regeneration module is used to regenerate or re-render only the target keyframes or target video segments that fail the gating detection, based on the parameter correction vector. The video output module is used to splice the video segments according to the transition type parameters after the video segments pass the gating detection, and to output the finished product by applying the 3D LUT parameters.

Images