A short video generation method and system based on multi-agent cooperation

Abstract

The present application relates to the technical field of video content generation, in particular to a short video generation method and system based on multi-agent cooperation, comprising: a director agent completes the analysis and conversion of the creation instruction to a structured screenplay; a camera movement agent decouples the camera motion instruction execution parameters to generate a motion control matrix; an asset agent builds a shared asset library and extracts role visual features and scene depth feature vectors therefrom; an audio and video synthesis agent feeds the motion control matrix into a diffusion model self-attention layer, and imports visual and depth features into a cross-attention layer, and outputs a storyboard segment after noise reduction rendering; a review agent judges the smoothness of the picture stream according to the motion vector field, and when the smoothness does not meet the requirements, calculates the partial derivative along each dimension of the matrix to fine-tune the attention weight and perform local re-rendering; and a audio and video agent splices the qualified segments and audio tracks to export a digital video stream. The smoothness of the camera movement and the consistency of the picture are improved.

Description

Technical Field

[0001] This invention relates to the field of video content generation technology, specifically to a short video generation method and system based on multi-agent collaboration. Background Technology

[0002] In recent years, short video platforms have developed rapidly, and AIGC technology has continued to mature. Automated short video generation using artificial intelligence has become a key trend in video content production. Video generation solutions based on multi-agent collaborative architectures have gained widespread attention in the industry because they can collaboratively handle tasks such as script analysis, visual rendering, and audio synthesis. However, the current field of AI short video generation still has the following shortcomings: First, camera motion control mainly relies on natural language descriptions in the storyboard text or indirect control signals in two-dimensional planes. It lacks precise parametric control methods for camera displacement, rotation, and focal length changes in three-dimensional space. This leads to coarse camera movement trajectories in the generated videos, specifically manifested in image shakiness, camera drift, and uneven camera speed, making it difficult to meet the quality requirements of short videos. Secondly, the content rendering of the video image and the camera motion trajectory are two independent processing links. The diffusion model lacks synchronous constraints from camera motion parameters to guide it during the denoising generation stage, resulting in insufficient coordination between the image content and the camera movement. This manifests as unnatural relative displacement between moving objects and the background, visual style drift during scene transitions, and difficulty in maintaining consistency in character appearance. Third, the methods for quality assessment are still rather rudimentary. Mainstream approaches rely on a discriminator providing a binary "pass / fail" conclusion, or on post-processing geometric alignment of generated frames, lacking quantitative measurement methods for the smoothness of motion between video frames. This leads to the problem that the system cannot accurately determine which frames or regions exhibit motion anomalies. For video clips with only localized defects, there are only two choices: regenerate the entire video or accept a lower-quality output, making it difficult to simultaneously guarantee generation efficiency and quality consistency. The technical problem this invention aims to solve is how to improve the synchronization and coordination between camera movement and scene content, as well as the smoothness of the video in AI-generated short videos.

[0003] To address this, a short video generation method and system based on multi-agent collaboration is proposed. Summary of the Invention

[0004] The purpose of this invention is to provide a short video generation method and system based on multi-agent collaboration. By relying on the cooperation and linkage between multiple agents, it can achieve fine-grained control of camera movement parameters, coordinated optimization of the two links of content generation and motion control, and closed-loop iterative improvement of video quality. This addresses the following technical difficulties in existing solutions: image shaking during camera movement, mismatch between lens movement and image content, and lack of targeted repair methods for local defects.

[0005] To achieve the above objectives, the present invention provides the following technical solution: A short video generation method based on multi-agent collaboration includes: A structured script containing storyboard descriptions is generated based on the initial creation instructions, and the camera motion instructions in the storyboard descriptions are converted into camera motion control matrices. Obtain the visual feature embedding vector and scene depth feature vector of the target character; send the camera motion control matrix into the self-attention mechanism layer of the diffusion model, and at the same time send the visual feature embedding vector and the scene depth feature vector into the cross-attention mechanism layer of the diffusion model to complete the denoising rendering and produce the storyboard video clip. The motion vector field of the storyboard video clip is obtained and its smoothness is evaluated. When the evaluation result is lower than a preset threshold, the partial derivatives of the evaluation score with respect to each dimension of the camera motion control matrix are calculated. The response weights of the camera motion control matrix in the self-attention mechanism layer are adjusted according to the partial derivatives and local re-rendering is performed to obtain a qualified storyboard video clip. The qualified storyboard video clip is then combined with the audio signal to export a digital video stream.

[0006] Preferably, the structured script is generated by a directing agent: the directing agent consists of a director sub-agent and a screenwriter sub-agent; firstly, the director sub-agent performs hierarchical decomposition of the video narrative text along the timeline and emotional curve to generate a narrative structure tree covering the scene level, scene level, and shot level; then, based on the narrative structure tree, the screenwriter sub-agent performs expansion and prompt word optimization operations on the text corresponding to each shot to form the structured script containing shot descriptions, character dialogue, and voice-over text.

[0007] Preferably, the camera motion control matrix is ​​obtained through a camera movement agent: extracting lens motion text and spatial orientation description from the shot description; then, calling a pre-trained semantic parsing model, which parses the lens motion text into the camera's translation trajectory and rotation angle in three-dimensional space, and simultaneously parses the spatial orientation description into focal length interpolation coefficients and focal length range; according to preset motion decoupling rules, decoupling the translation trajectory into three-dimensional spatial coordinates with axial translation, decoupling the rotation angle into pitch angle, yaw angle, and roll angle, and decoupling the focal length interpolation coefficients and focal length range into focal length variation parameters; finally, discretizing the decoupled three-dimensional spatial coordinates, pitch angle, yaw angle, roll angle, and focal length variation parameters according to a preset time series sampling rate to construct the time-series mapped camera motion control matrix.

[0008] Preferably, the process of obtaining the visual feature embedding vector and the scene depth feature vector specifically involves: obtaining character attribute text and scene environment text from the structured script; generating a set of reference images for the target character under different perspectives and postures based on the character attribute text; calling a preset image encoder to extract cross-view shared features from the reference image set; generating the visual feature embedding vector that can remain consistent across shots through mapping; generating a three-dimensional scene mesh or multi-view depth map based on the scene environment text; using a depth estimation and geometric feature extraction network to obtain the spatial relative distance distribution of the scene; constructing the scene depth feature vector; and associating and binding the visual feature embedding vector, the scene depth feature vector, and the corresponding asset identifier together and storing them in a global shared asset library.

[0009] Preferably, the diffusion model process is as follows: the camera motion control matrix is ​​encoded into a three-dimensional spatiotemporal attention mask and injected into the self-attention mechanism layer of each downsampling and upsampling module of the diffusion model to constrain the motion trajectory of the camera between video frames; in the parallel processing path, the visual feature embedding vector and the scene depth feature vector are concatenated and then feature projected, and input into the cross-attention mechanism layer of the diffusion model in the form of key vector and value vector, and cross-attention is calculated with the query vector of potential image features to guide the generation of screen content and scene geometry during the denoising rendering process.

[0010] Preferably, the process of producing the storyboard video clip specifically includes: generating audio with lip-sync timestamps based on the character dialogue in the structured script, and matching background music and sound effects based on the storyboard emotional tags; using an audio-video alignment model, fine-tuning the re-rendered frames of the character's lips in the qualified storyboard video clip according to the lip-sync timestamps; performing multi-track synthesis and encoding resampling on the aligned qualified storyboard video clip, audio, background music, and sound effects, and exporting the digital video stream.

[0011] Preferably, the qualified storyboard video clip is processed by the review agent by performing the following operations: first, using optical flow estimation to extract pixel motion vectors between adjacent frames in the storyboard video clip, constructing the motion vector field; calculating the temporal motion smoothness score and stuttering coefficient based on the motion vector field, and evaluating the smoothness assessment result; when the smoothness assessment result is lower than a preset threshold, locating the video frame region with abnormal motion, and using the smoothness assessment result as the evaluation score to construct a loss function for each dimension parameter of the camera motion control matrix; using the backpropagation algorithm to calculate the partial derivative matrix of the loss function with respect to the three-dimensional spatial coordinates, pitch angle, yaw angle, roll angle, and focal length change parameters in the camera motion control matrix; adjusting the attention weight coefficients injected into the self-attention mechanism layer of the camera motion control matrix according to the partial derivative matrix, performing local denoising and re-rendering on the video frame region with abnormal motion, and generating the qualified storyboard video clip.

[0012] A short video generation system based on multi-agent collaboration includes: The directing and camera movement agent generates a structured script containing storyboard descriptions based on initial creation instructions, and converts the camera motion instructions in the storyboard descriptions into a camera motion control matrix; the audio-visual synthesis agent obtains the visual feature embedding vector and scene depth feature vector of the target character; the camera motion control matrix is ​​fed into the self-attention mechanism layer of the diffusion model, and the visual feature embedding vector and the scene depth feature vector are fed into the cross-attention mechanism layer of the diffusion model to complete denoising rendering and produce storyboard video clips; the review agent calculates the motion vector field of the storyboard video clips and performs smoothness evaluation. When the evaluation result is lower than a preset threshold, the partial derivatives of the evaluation score with respect to each dimension of the camera motion control matrix are calculated. Based on the partial derivatives, the response weights of the camera motion control matrix in the self-attention mechanism layer are adjusted and local re-rendering is performed to obtain qualified storyboard video clips; The output agent combines the qualified storyboard video clips with the audio signal to export a digital video stream.

[0013] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. By coordinating the directing agent and the camera movement agent, the storyboard in natural language form is systematically converted into a camera motion control matrix that includes three-dimensional spatial coordinates, pitch angle, yaw angle, roll angle, and focal length change parameters. Compared with the existing approach of relying on text description to indirectly control the camera movement, the level of precision and smoothness of the camera movement trajectory are improved, which has a positive effect on reducing image shake and camera movement drift in the generated video.

[0014] 2. By constructing a global shared asset library through an asset intelligence agent, the embedding vectors of character visual features and scene depth feature vectors are extracted. The camera motion control matrix and the above two types of feature vectors are injected into the self-attention layer and cross-attention layer of the diffusion model for joint denoising and rendering. This allows the video frame generation process to synchronously reference camera motion parameters and scene geometric information. The matching degree of the picture content and lens movement between the storyboard segments and the consistency of the character visual style are improved compared to the method of generating each storyboard independently.

[0015] 3. By reviewing the intelligent agent to perform a smoothness quantification evaluation of the storyboard video clips based on the optical flow motion vector field, and performing gradient backpropagation along each dimension of the camera motion control matrix when the evaluation fails to meet the standard, attention weights are adjusted and local re-rendering is performed. This allows the abnormal motion frame regions in the video to be located and repaired in a targeted manner, reducing the number of times the entire video is regenerated. Compared with the method of making binary judgments based solely on the discriminator, this method improves both generation efficiency and the targeted nature of local quality repair. Attached Figure Description

[0016] Figure 1 This is a flowchart of a short video generation method based on multi-agent collaboration proposed in this invention. Figure 2 This is a flowchart of a short video generation method based on multi-agent collaboration proposed in this invention. Figure 3 This is a system architecture diagram of a short video generation system based on multi-agent collaboration proposed in this invention. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] Please see Figures 1 to 2 This invention provides a short video generation method based on multi-agent collaboration, the technical solution of which is as follows: A short video generation method based on multi-agent collaboration, such as Figures 1-2 As shown, it includes: A structured script containing storyboard descriptions is generated based on the initial creation instructions, and the camera motion instructions in the storyboard descriptions are converted into camera motion control matrices. Obtain the visual feature embedding vector and scene depth feature vector of the target character; send the camera motion control matrix into the self-attention mechanism layer of the diffusion model, and at the same time send the visual feature embedding vector and the scene depth feature vector into the cross-attention mechanism layer of the diffusion model to complete the denoising rendering and produce the storyboard video clip. The motion vector field of the storyboard video clip is obtained and its smoothness is evaluated. When the evaluation result is lower than a preset threshold, the partial derivatives of the evaluation score with respect to each dimension of the camera motion control matrix are calculated. The response weights of the camera motion control matrix in the self-attention mechanism layer are adjusted according to the partial derivatives and local re-rendering is performed to obtain a qualified storyboard video clip. The qualified storyboard video clip is then combined with the audio signal to export a digital video stream.

[0019] The initial creation instruction can be a video narrative text described in natural language, such as "A traveler walks through a bamboo forest at dusk; the camera slowly zooms in from a long shot to a medium shot, a gentle breeze rustles the bamboo leaves." Regarding the diffusion model, the backbone network uses a Stable Diffusion Video architecture, with a latent space encoding dimension of [missing information]. Set as The network consists of four downsampling modules and four upsampling modules. Both types of modules have a self-attention mechanism layer and a cross-attention mechanism layer. The preset threshold is the fluency threshold. The value is determined as follows: the optical flow feature distribution of camera movement segments in the publicly available video dataset is statistically analyzed, and the 10th percentile of the motion smoothness score distribution is used as the lower limit of the threshold; the initial value of the evaluation score S is taken from the temporal motion smoothness score in the smoothness evaluation result. With the normalized Karnökyi abrupt change coefficient Weighted sum: ; In the formula , In local re-rendering, the "local" range is defined as extending forward and backward from the video frame with abnormal motion as the center. Frames, totaling The time window formed by the frames.

[0020] Furthermore, the structured script is generated by a director-director intelligence agent: the director-director intelligence agent consists of a director sub-intelligence agent and a screenwriter sub-intelligence agent; firstly, the director sub-intelligence agent performs hierarchical decomposition of the video narrative text along the timeline and emotional curve to generate a narrative structure tree covering the scene level, the scene level, and the shot level; then, based on the narrative structure tree, the screenwriter sub-intelligence agent performs expansion and prompt word optimization operations on the text corresponding to each shot to form the structured script containing shot descriptions, character dialogue, and voice-over text.

[0021] The director sub-agent is based on the pre-trained large language model Llama 3.1-70B-Instruct, and guided by cue word engineering, it performs hierarchical decomposition of the video narrative text according to the following rules: Scene hierarchy is used to detect keywords in the text that indicate scene transitions, such as "at the same time," "a few days later," and "a scene change," and to divide the text into several narrative scenes in combination with plot turning points; The scene hierarchy is further subdivided into several scenes within each scene based on anchor words of spatial location, such as "indoor", "outdoor", and "mountain top". The storyboard hierarchy is used to further divide each scene into several scenes based on the rhythmic breaks in the action, such as semantic pauses marked by periods and semicolons.

[0022] The sentiment curve is generated by a pre-trained sentiment analysis model that scores each scene segment in the text using 28 sentiment distributions. The top three sentiment dimensions with the highest scores are then used to form a three-dimensional sentiment label sequence. ; Cubic spline interpolation is used to connect the emotional labels of adjacent storyboards into a smooth emotional curve. The screenwriter sub-agent is also built on Llama 3.1-70B-Instruct. During the expansion stage, it adds environmental details, character actions, and dialogue content to each storyboard description. During the prompt word optimization stage, it converts the expanded text into storyboard description field items in structured JSON format to meet the input needs of subsequent camera movement agents and asset agents.

[0023] The narrative text is decomposed into three layers—scene, scene, and storyboard—by the director sub-agent along the timeline and emotional curve. Then, the screenwriter sub-agent expands the storyboard text and optimizes the prompts. The entire process of transitioning from the initial creative instructions to a structured script that includes shot descriptions, character dialogue, and voice-over no longer requires human intervention. Compared to manual storyboarding or automatic storyboarding that relies solely on text segmentation, the storyboard structure improves both narrative rationality and emotional rhythm matching.

[0024] Furthermore, the camera motion control matrix is ​​obtained through a camera movement agent: lens motion text and spatial orientation description are extracted from the scene description; then, a pre-trained semantic parsing model is invoked, which parses the lens motion text into the camera's translation trajectory and rotation angle in three-dimensional space, and simultaneously parses the spatial orientation description into focal length interpolation coefficients and focal length range; according to preset motion decoupling rules, the translation trajectory is decoupled into three-dimensional spatial coordinates with axial translation, the rotation angle is decoupled into pitch angle, yaw angle, and roll angle, and the focal length interpolation coefficients and focal length range are decoupled into focal length variation parameters; finally, the decoupled three-dimensional spatial coordinates, pitch angle, yaw angle, roll angle, and focal length variation parameters are discretized according to a preset time-series sampling rate to construct the time-series mapped camera motion control matrix.

[0025] The pre-trained semantic parsing model in this paper adopts a text-to-motion parameter model based on the T5-base architecture, and is trained on a self-constructed camera motion description-motion parameter pairing dataset with a set learning rate. Batch size The model was fine-tuned and trained for 3 epochs. The input to the model is a sequence formed by concatenating camera motion text and spatial orientation description, and the output is a continuous sequence of motion parameters.

[0026] The specific details of the preset motion decoupling rules are as follows: The translation trajectory output by the semantic parsing model is denoted as: ; The rotation angle is denoted as: ; Focal length interpolation coefficient is The focal length range is ,in For storyboard timing parameters, The duration of this scene (s). The decoupled three-dimensional spatial coordinates. Established with the camera's initial position as the origin of the coordinate system. In the three-dimensional camera coordinate system, the positive direction of the lens optical axis is defined as... Positive axis, horizontal to the right is Positive axis, vertically upward. Positive axis: ; Pitch angle Yaw angle Roll angle Corresponding to the circumference axis, axis, Euler angle rotational components of the axis, in radians: ; Rotation sequence adopted Euler angle convention, that is, first around The axis rotates and rolls at an angle, then around... The axis rotates at the yaw angle, and finally around the yaw angle. The axis of rotation pitch angle.

[0027] Focal length variation parameters Determined by linear interpolation of focal length interpolation coefficients and focal length range: ; in, and These focal lengths are derived from the ranges given in the spatial orientation description, with typical values ​​ranging from [missing information]. , , This represents the normalized focal length interpolation coefficient.

[0028] The time series sampling rate is set to The frame rate is kept consistent with the target video frame rate. The seven parameters obtained from decoupling... ,by Let be the time step, at discrete time... ( Sampling is performed on the sampled area, with each sampling point arranged in a column to form a sample. 3D camera motion control matrix : ; in .

[0029] Most existing solutions rely on text labels or two-dimensional planar displacement to indirectly describe camera movement. However, this embodiment transforms the camera trajectory into a seven-dimensional parameter matrix that covers spatial coordinates, rotation angles, and focal length changes. The improvement in multi-dimensional accuracy directly leads to the improvement in camera movement stability, and image shake and drift are also reduced.

[0030] Furthermore, the process of obtaining the visual feature embedding vector and the scene depth feature vector specifically involves: obtaining character attribute text and scene environment text from the structured script; generating a set of reference images for the target character under different perspectives and poses based on the character attribute text; calling a preset image encoder to extract cross-view shared features from the reference image set; generating the visual feature embedding vector that can remain consistent across shots through mapping; generating a three-dimensional scene mesh or multi-view depth map based on the scene environment text; using a depth estimation and geometric feature extraction network to obtain the spatial relative distance distribution of the scene; constructing the scene depth feature vector; and then associating and binding the visual feature embedding vector, the scene depth feature vector, and the corresponding asset identifier together and storing them in the global shared asset library.

[0031] For the image encoder, the CLIP-ViT-L / 14 model was selected, which adopts the Vision Transformer architecture. Pixels, the output of the last layer of ViT As a global representation of the image, it is then passed through a fully connected projection matrix. Mapped to The result of the 3D feature space is the visual feature embedding vector.

[0032] The reference image set was generated as follows: for each target character, using the character's attribute text as input, the Stable Diffusion XL model was called, and eight directional viewpoint cues were overlaid. The posture cues include orientation cues such as "straight ahead" and "left front". Left, Left Back Rear right, right rear Right, front right Posture cues included standing, walking, and sitting postures; these were combined to generate 24 reference images. The ViT final layer of each of the 24 reference images was then extracted. Features are then analyzed and mean pooled to obtain 768-dimensional cross-view shared features. Then it passes through a fully connected projection matrix. Mapped to The result of this dimensional feature space is the visual feature embedding vector. .

[0033] The backbone of the depth estimation and geometric feature extraction network is DPT-Large, which takes a single-frame scene image corresponding to the scene environment text as input and outputs a pixel-level depth map. Preset maximum perception depth Then, the depth map is sorted according to... The grid is divided, and the average depth value within each cell is taken to obtain... A coarse geometric descriptor for the dimension; simultaneously inputting the depth map based on Feature extraction branch, output Fine-grained geometric features of the dimension. The above coarse geometric descriptor is concatenated with the fine-grained geometric features and compressed through a fully connected layer to the desired size. Dimension, construct the scene depth feature vector The scene depth feature vector is a dimensionless vector, and each component of the vector reflects the normalized relative depth value of different spatial sub-regions relative to the camera: ; in For the first The average depth of each sub-region This is the minimum effective depth.

[0034] The asset identifier adopts a hierarchical encoding format: {role / scene ID}-{asset type}-{version number}. For example, CH001-VISUAL-v1 represents the first version of the visual embedding vector of the role numbered CH001, and SC005-DEPTH-v2 represents the second version of the scene depth feature vector of the role numbered SC005.

[0035] With the help of the asset library mechanism mentioned above, the appearance and spatial geometry of the same character or scene can maintain consistency across shots, thereby effectively alleviating the problems of visual style abrupt changes and character image mismatch that are prone to occur in the shot switching process.

[0036] Furthermore, the diffusion model process is as follows: the camera motion control matrix is ​​encoded into a three-dimensional spatiotemporal attention mask and injected into the self-attention mechanism layer of each downsampling and upsampling module of the diffusion model to constrain the camera's motion trajectory between video frames; in the parallel processing path, the visual feature embedding vector and the scene depth feature vector are concatenated and then feature projected, and input into the cross-attention mechanism layer of the diffusion model in the form of key vector and value vector, and cross-attention is calculated with the query vector of potential image features to guide the generation of screen content and scene geometry during the denoising rendering process.

[0037] Camera motion control matrix The process of encoding into a 3D spatiotemporal attention mask is as follows: for A dimensional matrix is ​​input into a time encoder consisting of two fully connected layers, with dimensions of 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19 ... Each column Mapped to 3D time-step embedding vector ,get The sequence is embedded in time steps. Subsequently, this sequence is processed by a self-attention temporal aggregation module to fuse the motion relationships between frames. This module employs a single-layer multi-head self-attention structure with a certain number of heads. Dimensions per head The final output is a spatiotemporal attention mask. ,in Indicates the first Frame and the The strength of motion correlation between frames. In the first... In the self-attention calculation of each downsampling or upsampling module, the original attention weights are calculated according to the inter-frame temporal token axis. Element-wise multiplication Modulate the weights and then re-normalize them row by row: The original attention weights Element-wise multiplication Modulation: ; in This represents element-wise multiplication. The modulation intensity coefficient, These are the query matrix, key matrix, and key dimension of self-attention, respectively.

[0038] In the parallel processing path, the visual features are embedded into the vector. With the scene depth feature vector By concatenating along the feature dimensions, we obtain ; mapped to a key vector matrix containing multiple sets of conditional tokens through a linear projection matrix. Sum value vector matrix Input cross-attention mechanism layer and latent feature dimension Complete weighted interaction; the self-attention path injected by camera motion and the cross-attention path injected by feature conditions are executed synchronously within the same denoising time step, and the total number of denoising time steps is [not specified]. ,use Sampling strategy.

[0039] This application establishes a synchronous constraint relationship between the camera motion trajectory and the denoising rendering process of the diffusion model, thereby improving the degree of coordination between the inter-frame lens motion and the generation of image content. The problems of disconnect between lens motion and image content or unnatural relative displacement between moving objects and the background are significantly reduced.

[0040] Furthermore, the process of generating the storyboard video clip specifically includes: generating audio with lip-sync timestamps based on the character dialogue in the structured script, and matching background music and sound effects based on the storyboard emotional tags; using an audio-video alignment model, fine-tuning the re-rendered frames of the character's lips in the qualified storyboard video clip according to the lip-sync timestamps; and performing multi-track synthesis and encoding resampling on the aligned qualified storyboard video clip, audio, background music, and sound effects to export the digital video stream.

[0041] The audio is generated by a Chinese speech synthesis model based on the VITS architecture. The input is the character dialogue text in a structured script, and the output is... Sampling rate, Quantized mono Audio. The lip-sync timestamp is generated by... The tool performs phoneme-level alignment on the synthesized speech, which is then converted into a time series of lip movement keyframes via a phoneme-visual mapping table. .

[0042] The emotional tags for the storyboard scenes use the three-element emotional tag sequence output by the director's intelligent agent. The primary emotional dimension in the scene serves as the overall emotional label for that scene, such as "calm," "tense," "joyful," or "sad." The background music library contains 10 sub-libraries corresponding to these emotional categories, with each sub-library containing 50 background music tracks. The matching process calculates the cosine similarity between the desired emotional vector and the pre-labeled emotional vectors of each track, selecting the top 3 tracks with the highest similarity as recommendations. Sound effect matching uses the same method; the sound effect library contains 200 common scene sound effects, including wind sounds, footsteps, and door hinge sounds, and is searched and matched based on keywords describing the scene's environment.

[0043] The audio-video alignment model is Wav2Lip-GAN. The input consists of the lip region of a character in a valid storyboard video clip and the corresponding audio segment. The output is the aligned video frame after local lip re-rendering. The multi-track synthesis uses a video track as the main track, audio as the second track, background music as the third track, and sound effects as the fourth track. Each audio track is configured with... After resampling at the sampling rate, amplitude normalization is performed; the RMS of the audio track is normalized to... Background music track normalized to Sound track normalized to The final encoding resampling adopts... Video encoding, bitrate Keyframe interval Frames, and Audio encoding, bitrate Encapsulated as Export digital video streams in container format.

[0044] The synchronization accuracy between character lip movements and voice audio in terms of lip-sync has been improved, the synthesis timing alignment between multi-track audio and video is tighter, and problems such as audio-visual asynchrony and mismatch between lip shape and pronunciation have been reduced.

[0045] Furthermore, the qualified storyboard video clip undergoes the following operations by the review agent: First, optical flow estimation is used to extract pixel motion vectors between adjacent frames in the storyboard video clip, constructing the motion vector field; based on the motion vector field, temporal motion smoothness score and stuttering coefficient are calculated respectively to obtain the smoothness evaluation result; when the smoothness evaluation result is lower than a preset threshold, the abnormal motion video frame region is located, and the smoothness evaluation result is used as the evaluation score to construct a loss function for each dimension parameter of the camera motion control matrix; the backpropagation algorithm is used to calculate the partial derivative matrix of the loss function with respect to the three-dimensional spatial coordinates, pitch angle, yaw angle, roll angle, and focal length change parameters in the camera motion control matrix; based on the partial derivative matrix, the attention weight coefficients injected into the self-attention mechanism layer of the camera motion control matrix are adjusted, and local denoising and re-rendering are performed on the abnormal motion video frame region to generate the qualified storyboard video clip.

[0046] The optical flow estimation method employs the RAFT algorithm, which, based on pre-trained RAFT-Sintel weights, performs calculations on adjacent frames. and Calculating dense optical flow fields Both frames are Resolution; Optical Flow Vector Represents pixels From the Frame to the Frame in direction and Displacement in direction, measured in pixels per frame.

[0047] The temporal motion smoothness score The calculation method is as follows: First, the optical flow field... The magnitude of the optical flow vector of all pixels is calculated. Then take the total The negative logarithmic mean of the rate of change of optical flow amplitude between adjacent frames within a frame segment: ; ; in To prevent smoothing terms that take the logarithm of zero. Smoothness score. This is a dimensionless relative score; the higher the value, the smoother the motion. The unit is pixels per frame, which represents the average change in optical flow amplitude between frames.

[0048] The coefficient of catastrophic change The calculation method is as follows: ; ; ; in This refers to pixel-level motion dynamism (the rate of change of optical flow between frames). For frame intervals, Let be the average jerk measure of frame t. Calculate the normalized stutter coefficient. ,in This is a reference normalization constant.

[0049] The evaluation score The weighted sum of the two indicators above: ; in Smoothness threshold --when The system was judged to be not smooth enough.

[0050] The strategy for locating the video frame region with abnormal motion is as follows: traverse all frames within the storyboard segment, and find those that satisfy... The frames are marked as motion abnormal frames, where , and These are the mean and standard deviation of the average jerkness across all frames within the storyboard segment. Consecutive aberrant frames are merged into aberrant frame segments, and the first and last values ​​of each segment are extended. The frame is defined as the region of the video frame with the motion anomaly.

[0051] The loss function L is defined as the product of the negative evaluation score within a motion-abnormal frame region and the number of frames within that region, to encourage the optimization process to improve motion smoothness within that region. ; in A set of frames with motion anomalies. The evaluation score is calculated frame by frame. and For the first Frame-centric Partial values ​​within the frame window For each dimension of the camera motion control matrix Regularization term, regularization coefficient .

[0052] Using the Adam optimizer, the learning rate... For camera motion control matrix Gradient descent optimization is performed in each dimension within the time period corresponding to the abnormal frame region, with an iteration upper limit. Step or early stop conditions Each iteration updates the attention weight coefficients: Let the first... The original attention weights of the layer are The adjusted weights are: ; in For the partial derivative matrix Normalized adjustment amount When performing local denoising and re-rendering on video frames with abnormal motion, the window boundary frames inherit the latent spatial features and noise states of neighboring normal frames to maintain visual continuity, only rendering frames belonging to the abnormal motion region. The denoising process is re-executed for frames within the extended window; denoising time steps. ,use Sampling is performed, and frames outside the window remain unchanged.

[0053] This embodiment can locate and perform targeted local repair on video frame regions with abnormal motion, thereby avoiding the redundant computational overhead caused by having to regenerate the entire video due to local lens defects, and improving both generation efficiency and the overall motion smoothness of the video clip.

[0054] In this application, multiple intelligent agents collaborate to complete the entire generation process from text to finished short video. Each processing step forms a closed loop through a differentiable parameter transfer mechanism, thus taking into account the stability of camera movement, the consistency of character vision, and the repairability of local quality defects.

[0055] Example 2 Please see Figure 3 This invention provides a short video generation system based on multi-agent collaboration, the technical solution of which is as follows: A short video generation system based on multi-agent collaboration, such as Figure 3 As shown, it includes: The directing and camera movement agent generates a structured script containing storyboard descriptions based on initial creation instructions, and converts the camera motion instructions in the storyboard descriptions into a camera motion control matrix; the audio-visual synthesis agent obtains the visual feature embedding vector and scene depth feature vector of the target character; the camera motion control matrix is ​​fed into the self-attention mechanism layer of the diffusion model, and the visual feature embedding vector and the scene depth feature vector are fed into the cross-attention mechanism layer of the diffusion model to complete denoising rendering and produce storyboard video clips; the review agent calculates the motion vector field of the storyboard video clips and performs smoothness evaluation. When the evaluation result is lower than a preset threshold, the partial derivatives of the evaluation score with respect to each dimension of the camera motion control matrix are calculated. Based on the partial derivatives, the response weights of the camera motion control matrix in the self-attention mechanism layer are adjusted and local re-rendering is performed to obtain qualified storyboard video clips; The output agent combines the qualified storyboard video clips with the audio signal to export a digital video stream.

[0056] Using the automatic generation of short promotional videos for ancient-style cultural tourism as an application background, the system described in this application is put into actual operation. The initial creation instruction received in this scenario is a narrative text that integrates classical dance and ancient architecture elements: Under the afterglow of the setting sun, a dancer dressed in a red and white Tang-style ruqun (a type of traditional Chinese dress) stands in the center of an ancient architectural square, dancing gracefully; the camera starts with a close-up of the dancer's face, slowly circling and pulling back along a smooth arc, finally settling on a panoramic view of the palace complex; a gentle breeze rustles the dancer's clothes, and the sounds of guzheng (a traditional Chinese zither) and wind chimes intertwine in the background. After receiving this initial creation instruction, the director sub-agent performs hierarchical decomposition of the narrative text based on a pre-trained large language model. By identifying scene transitions and spatiotemporal nodes, the director sub-agent breaks down the text into a narrative structure tree, with the scene level positioned as the ancient palace under the afterglow of the setting sun, the field level focusing on the dance floor in the center of the square, and the shot level specifically defined as a close-up circling shot of the face in shot one and a panoramic zoom-out shot in shot two. Meanwhile, the pre-trained sentiment analysis model scores the text, producing a triadic sentiment tag sequence consisting of elegance, nostalgia, and grandeur. The screenwriter sub-agent then expands the storyboard text, adding details such as the texture of the Tang Dynasty dress, the transitions between light and shadow, and specific depictions of dance movements. This is then converted into a structured JSON-formatted storyboard description file, ultimately outputting a structured script that includes details of the shots, character dialogue, and sound effects tags.

[0057] The camera movement agent extracts camera movement text (smooth zooming and panning) and spatial orientation descriptions (from close-ups of faces to panoramic views of palaces) from the structured script. It then calls upon a pre-trained T5-based semantic parsing model to parse the camera movement text into arc-shaped translation trajectories in three-dimensional space. With rotation angle Furthermore, the spatial orientation description is analyzed as the focal length change rate and focal length range. Based on the preset motion decoupling rules, the translation trajectory is decoupled into three-dimensional spatial coordinates with axial translation. Decouple the rotation angle into the pitch angle. Yaw angle and roll angle Decoupling the focal length range from Corresponding wide-angle panorama to Corresponding focal length variation parameters for close-up portraits according to The time series sampling rate for the above Discretized sampling was performed on parameters in each dimension, and sampling was performed for a 5-second segment. Data from each time step is used to construct a data set of size [size missing]. Timing-mapped camera motion control matrix .

[0058] The asset agent, based on the character attribute text in the structured script—namely, a dancer dressed in a red and white Tang-style ruqun (a type of traditional Chinese dress)—uses the Stable Diffusion XL model to generate a set of 24 reference images covering 8 viewpoints and 3 dance poses. Cross-view shared features are extracted using the CLIP-ViT-L / 14 image encoder, and then projected to generate a 512-dimensional visual feature embedding vector. This ensures consistency between dancers' costumes and facial features across shots. Simultaneously, the asset agent, based on the scene environment text—namely, the ancient architectural plaza bathed in the afterglow of sunset—uses a DPT-Large depth estimation network and a PointNet++ geometric feature extraction network to extract the spatial relative distance distribution of the plaza and palace complex, constructing a 512-dimensional scene depth feature vector. .Will and After being bound to asset identifiers CH002-VISUAL-v1 and SC009-DEPTH-v1, it is stored in the global shared asset library.

[0059] The audio / video synthesis agent injects camera motion control matrices and asset features into the diffusion model. First, the audio / video synthesis agent... 3D camera motion control matrix Encoded using a two-layer fully connected temporal encoder and a self-attention temporal aggregation module, as follows: 3D spatiotemporal attention mask This is injected into the self-attention mechanism layer of each downsampling and upsampling module in the SVD backbone network, so as to... The modulation intensity constrains the inter-frame camera's arc-shaped wraparound and zoom-out trajectory. Visual features are embedded into vectors in the parallel processing path. With scene depth feature vector The features are concatenated into 1024 dimensions, projected, and then input into the cross-attention mechanism layer of the diffusion model as key and value vectors. Denoising rendering is performed using the DDIM sampling strategy within 50 denoising time steps, producing an initial storyboard video clip containing 120 frames.

[0060] After completing the denoising rendering of the initial storyboard video clips, the review agent invokes the RAFT algorithm to extract pixel motion vectors between adjacent frames and establish a motion vector field. By calculating smoothness and stuttering scores, the initial smoothness score was found to be below a specified lower limit. Using the jerk threshold The screening process identified an abnormal camera movement at the transition point between the curved shot and the wide shot – a sudden change in trajectory and localized jitter occurred between frames 45 and 52. The reviewing agent extended outwards by four frames from this abnormal segment, classifying frames 41 to 56 as a unified category. Area. Regarding the repair strategy, first... Construct a loss function L, and find the pair of L's values. The partial derivatives of each parameter of the matrix are then used to feed the partial derivative information back to the weight coefficients of the self-attention layer. The correction was made, and DDIM denoising was finally re-executed only within a local time window of frames 41-56. The smoothness score improved after the fix. If the value exceeds the 0.85 threshold, the fragment quality meets the standard and can be used as a qualified output for subsequent synthesis.

[0061] The audio-visual synthesis agent synthesizes qualified storyboard video clips with audio signals across multiple tracks. Based on the sound effects and music tags in the structured script, it retrieves and matches the guzheng solo audio with the highest cosine similarity from the background music library as background music, and retrieves the sounds of wind and flowing clothing as environmental sound effects. Based on the lip-sync and timestamp alignment module, it uses the Wav2Lip-GAN model to fine-tune the local re-rendered frames of the dancers' lip movements for the dancers' backing vocals or narration lines in the storyboard. Finally, it performs multi-track time-axis alignment and synthesis with the background music track (normalized to -22dBFS) and the environmental sound effect track (normalized to -18dBFS) after 48kHz resampling, and encapsulates and exports the video using H.264 video encoding and AAC audio encoding to produce a high-definition MP4 digital video stream.

[0062] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A short video generation method based on multi-agent collaboration, characterized in that, include: A structured script containing storyboard descriptions is generated based on the initial creation instructions, and the camera motion instructions in the storyboard descriptions are converted into camera motion control matrices. Obtain the visual feature embedding vector and scene depth feature vector of the target character; send the camera motion control matrix into the self-attention mechanism layer of the diffusion model, and at the same time send the visual feature embedding vector and the scene depth feature vector into the cross-attention mechanism layer of the diffusion model to complete the denoising rendering and produce the storyboard video clip. The motion vector field of the storyboard video clip is obtained and its smoothness is evaluated. When the evaluation result is lower than a preset threshold, the partial derivatives of the evaluation score with respect to each dimension of the camera motion control matrix are calculated. The response weights of the camera motion control matrix in the self-attention mechanism layer are adjusted according to the partial derivatives and local re-rendering is performed to obtain a qualified storyboard video clip. The qualified storyboard video clip is then combined with the audio signal to export a digital video stream.

2. The short video generation method based on multi-agent collaboration according to claim 1, characterized in that: The structured script is generated by a directing agent: the directing agent consists of a director sub-agent and a screenwriter sub-agent; firstly, the director sub-agent performs hierarchical decomposition of the video narrative text along the timeline and emotional curve to generate a narrative structure tree covering the scene level, scene level, and shot level; then, based on the narrative structure tree, the screenwriter sub-agent performs expansion and prompt word optimization operations on the text corresponding to each shot to form the structured script containing shot descriptions, character dialogue, and voice-over text.

3. The short video generation method based on multi-agent collaboration according to claim 1, characterized in that, The camera motion control matrix is ​​obtained through a camera movement agent: Lens motion text and spatial orientation description are extracted from the scene description; then, a pre-trained semantic parsing model is invoked, which parses the lens motion text into the camera's translation trajectory and rotation angle in three-dimensional space, and simultaneously parses the spatial orientation description into focal length interpolation coefficients and focal length range; according to preset motion decoupling rules, the translation trajectory is decoupled into three-dimensional spatial coordinates with axial translation, the rotation angle is decoupled into pitch angle, yaw angle, and roll angle, and the focal length interpolation coefficients and focal length range are decoupled into focal length variation parameters; finally, the decoupled three-dimensional spatial coordinates, pitch angle, yaw angle, roll angle, and focal length variation parameters are discretized according to a preset time-series sampling rate to construct the time-series mapped camera motion control matrix.

4. The short video generation method based on multi-agent collaboration according to claim 1, characterized in that, The process of obtaining the visual feature embedding vector and scene depth feature vector is as follows: obtain the character attribute text and scene environment text from the structured script; generate a set of reference images for the target character under different perspectives and postures based on the character attribute text; call a preset image encoder to extract cross-view shared features from the set of reference images; and generate the visual feature embedding vector that can remain consistent across shots through mapping. A 3D scene mesh or multi-view depth map is generated based on the scene environment text. A depth estimation and geometric feature extraction network is used to obtain the spatial relative distance distribution of the scene and construct the scene depth feature vector. The visual feature embedding vector, the scene depth feature vector and the corresponding asset identifier are associated and bound together and stored in the global shared asset library.

5. The short video generation method based on multi-agent collaboration according to claim 1, characterized in that, The diffusion model process is as follows: the camera motion control matrix is ​​encoded into a three-dimensional spatiotemporal attention mask and injected into the self-attention mechanism layer of each downsampling and upsampling module of the diffusion model to constrain the camera's motion trajectory between video frames; in the parallel processing path, the visual feature embedding vector and the scene depth feature vector are concatenated and then feature projected, and input into the cross-attention mechanism layer of the diffusion model in the form of key vector and value vector, and cross-attention is calculated with the query vector of potential image features to guide the generation of screen content and scene geometry during the denoising rendering process.

6. The short video generation method based on multi-agent collaboration according to claim 1, characterized in that, The process of producing the storyboard video clip specifically includes: generating audio with lip-sync timestamps based on the character dialogue in the structured script, and matching background music and sound effects based on the storyboard emotional tags; using an audio-video alignment model, fine-tuning the re-rendered frames of the character's lips in the qualified storyboard video clip according to the lip-sync timestamps; performing multi-track synthesis and encoding resampling on the aligned qualified storyboard video clip, audio, background music, and sound effects, and exporting the digital video stream.

7. The short video generation method based on multi-agent collaboration according to claim 1, characterized in that, The qualified storyboard video clip undergoes the following operations by the review agent: First, optical flow estimation is used to extract pixel motion vectors between adjacent frames in the storyboard video clip, constructing the motion vector field; based on the motion vector field, temporal motion smoothness score and stuttering coefficient are calculated to obtain the smoothness evaluation result; when the smoothness evaluation result is lower than a preset threshold, the abnormal motion video frame region is located, and the smoothness evaluation result is used as the evaluation score to construct a loss function for each dimension parameter of the camera motion control matrix; the backpropagation algorithm is used to calculate the partial derivative matrix of the loss function with respect to the three-dimensional spatial coordinates, pitch angle, yaw angle, roll angle, and focal length change parameters in the camera motion control matrix; based on the partial derivative matrix, the attention weight coefficients injected into the self-attention mechanism layer of the camera motion control matrix are adjusted, and local denoising and re-rendering are performed on the abnormal motion video frame region to generate the qualified storyboard video clip.

8. A short video generation system based on multi-agent collaboration, characterized in that, include: The directing and camera movement agent generates a structured script containing storyboard descriptions based on initial creation instructions, and converts the camera motion instructions in the storyboard descriptions into a camera motion control matrix; the audio-visual synthesis agent obtains the visual feature embedding vector and scene depth feature vector of the target character; the camera motion control matrix is ​​fed into the self-attention mechanism layer of the diffusion model, and the visual feature embedding vector and the scene depth feature vector are fed into the cross-attention mechanism layer of the diffusion model to complete denoising rendering and produce storyboard video clips; the review agent calculates the motion vector field of the storyboard video clips and performs smoothness evaluation. When the evaluation result is lower than a preset threshold, the partial derivatives of the evaluation score with respect to each dimension of the camera motion control matrix are calculated. Based on the partial derivatives, the response weights of the camera motion control matrix in the self-attention mechanism layer are adjusted and local re-rendering is performed to obtain qualified storyboard video clips; The output agent combines the qualified storyboard video clips with the audio signal to export a digital video stream.

Images