Graph-to-video generation method based on untrained strategy and multi-agent attention alignment

Abstract

This invention discloses a graph-based video generation method based on a training-free strategy and multi-subject attention alignment, belonging to the field of graph-based video technology. This invention separates different subjects through subject perception and attention decoupling. When calculating attention, only the information of one subject is aligned, while the other subjects are masked. Attention calculation requires aligning the frame features and text features of each subject. Decoupling allows each subject to be extracted and aligned with its corresponding text features. Furthermore, other objects are not involved in the calculation, ensuring that each subject only exhibits actions corresponding to its own text in the final result.

Description

Technical Field

[0001] This invention relates to the field of graph-based video technology, and more specifically to a graph-based video generation method based on a training-free strategy and multi-agent attention alignment. Background Technology

[0002] Diffusion models have achieved remarkable success in the generative domain, especially in image generation, even surpassing Generative Adversarial Networks (GANs). With advancements in text-to-image generation techniques, video generation tasks have also seen rapid development. The Video Diffusion Model (VDM) is the first to extend the 2D U-Net to a 3D U-Net structure, enabling joint training of images and videos. Furthermore, AnimateDiff trains a motion module to adapt to different personalized text-to-image (T2I) models, combining it with other specialized content models to generate high-quality videos. Text2VideoZero proposes a sampling method that requires no additional training, enhancing motion dynamics while maintaining frame consistency, thereby generating video content that meets expectations.

[0003] Custom video generation aims to generate highly personalized video content based on specific user needs or input conditions, primarily focusing on the text-to-video domain. This is achieved by fine-tuning the generation of video subjects using text with specific symbols, and fine-tuning specific motions using text representing motion in the relevant video. The resulting video from fine-tuned text generation has the desired subjects and motions, thus achieving custom video generation. Existing custom video generation employs fine-tuning methods. For example, CustomVideo uses the DreamBooth fine-tuning method, modulating its model with specific text representing images. LAMP fine-tunes specific text actions, binding these actions to actions in the training video to achieve specific motion fine-tuning. DreamVideo fine-tunes spatial subjects and temporal actions separately to generate video models with specific subjects and actions. Recent research has explored multi-objective customization. DisenStudio takes image data containing multiple subjects as input and stabilizes the diffusion, fine-tunes specific text cues, and uses masks to distinguish different subjects, achieving text-driven motion for different subjects. Customized video generation has driven the development of multi-objective control in terms of subject and motion control, enriching the field of video generation.

[0004] Recently, research on multi-agent generation has made some progress. In the field of image generation, Be Yourself significantly alleviates the semantic confusion and misalignment problems of multiple agents in text during stable diffusion generation in a training-free manner. Mastering text-to-image diffusion utilizes a large language model to automatically separate agents in the text, forming individual cues. Each cue simultaneously enters the UNet network, and the final potential results are concatenated according to specific dimensions, ensuring that each agent does not interfere with the others, thus achieving multi-agent image generation. In the video domain, multi-agent generation is also rapidly developing, mainly focusing on personalized custom video generation. For example, DisenStudio and CustomVideo train their models by concatenating multiple agents into a single image. They use special placeholders to bind agents and use masks to distinguish different agents. The difference is that the former uses a concatenation method to combine the cross-attention of different agents based on different texts instead of the original attention and uses masks to distinguish them one by one. The latter uses masks to distinguish across the entire cross-attention.

[0005] Recent research in the field of image-to-video has made progress in the controllable motion of multiple subjects, such as Follow-Your-Pose v2, which animates multiple targets in an image based on multiple person poses. Image-to-video typically animates an input image based on given conditions to achieve dynamic content derived from the image. With the development of image-to-video, several branches have emerged. For example, text-guided image-to-video generation aims to generate videos that match both the text semantics and the image content. Human pose-guided models convert human images into videos with additional controls, such as dense poses, depth maps, etc. Image animation under optical flow conditions is similar to human pose-controlled animation, primarily focusing on character animation. Models that control motion in mask regions achieve this by controlling the motion of selected regions in the image, while in the trajectory control domain, they control the direction of motion. These models typically combine a clean image with initial noise or inject image information into initial noise before generating video through a U-Net network, ensuring consistency of spatial content while also focusing on motion in the temporal dimension.

[0006] The methods described above have achieved good performance, but there is still no research on using only text to drive image-generated video, and some key issues need to be addressed.

[0007] (1) In studies on image-to-video generation that rely solely on text, when the generated content involves multiple subjects, the content and motion of these subjects often become chaotic, making it difficult to accurately align the text with the corresponding subjects or distinguish between different subjects.

[0008] (2) In real-world scenarios, multi-subject images occur far more frequently than single-subject images. However, existing methods generally ignore this, which greatly limits the application of image-to-video generation in multi-subject scenarios. In addition, most existing video generation datasets are designed for scene motion or single-subject actions, lacking purely multi-subject generation datasets, and dataset creation and searching are also quite difficult. Summary of the Invention

[0009] The technical problem to be solved by this invention is: how to overcome the shortcomings of existing image-to-video methods in multi-subject scenarios. It provides an image-to-video generation method based on training-free strategy and multi-subject attention alignment. This method can not only accurately identify different subjects in the image and accurately match them with their respective text descriptions, but also realize that each subject can independently perform different actions, enriching the application scenarios of image-to-video generation and expanding the technical boundaries of this field.

[0010] The present invention solves the above-mentioned technical problems through the following technical solution, and the present invention includes the following steps:

[0011] S1: Get the input text and image;

[0012] S2: Separate the input text and segment the input image to obtain the mask of each subject, and perform perceptual processing on the mask;

[0013] S3: Input the extracted mask, the separated text, and the corresponding input image into the pre-trained image-to-video model based on the diffusion model;

[0014] S4: Encode the image, then copy it multiple times, and add noise to create additional noise;

[0015] S5: The separated text is fed into the CLIP text encoder to extract the features of each subject, and each subject feature is copied to the same number of frames as the target generated video. These copied subject features are then stored in the list set p.

[0016] S6: Input the obtained set p, the mask of each main region, and the noise into the U-Net network;

[0017] S7: When computing cross-attention in the U-Net network, a mask is used to obtain the feature q of each subject from multiple subject features q. l Main feature q l Matching text features k l and v l First, calculate q using the dot product. l With k lThe similarity between inputs is used to obtain an attention weight, representing the relevance between each input and other inputs. Then, the attention weights are normalized, and finally, these normalized attention weights are applied to the corresponding value vectors to generate the attention output, where k... l and v l Obtained from set p, where l represents the l-th subject;

[0018] S8: Concatenate multiple attention outputs and normalize the sum of all masks to obtain an attention DeAttention that matches the pre-trained image-to-video model based on the diffusion model. Replace the attention in the original attention module in the U-Net network, and then obtain the latent variable for denoising.

[0019] S9: Using a one-step denoising method, the classifier-free guidance method is used to generate the final latent variables and obtain the latent representation;

[0020] S10: By repeating the denoising process T times, noise is gradually eliminated to obtain the target generated video.

[0021] Furthermore, in step S2, the specific processing procedure is as follows:

[0022] S21: By separating the input text, the descriptive text containing multiple subjects is separated, and the description of each subject is stored in a separate set;

[0023] S22: Using the separated text description, segment each subject in the image through visual segmentation processing to generate the corresponding subject region mask;

[0024] S23: Then perform perception processing on each mask: In each mask, set the pixel value of the corresponding subject area to 1, and set the pixel value of other subjects area to 0, so that each mask accurately represents the area of ​​its corresponding subject.

[0025] Furthermore, in step S4, the specific processing procedure is as follows:

[0026] S41: After the input image is transformed into latent variables by a pre-trained encoder, it is then subjected to noise processing.

[0027] S42: Copy the latent variables multiple times, making the number of copies the same as the number of frames in the target generated video;

[0028] S43: Add noise of one frame dimension to each latent variable;

[0029] S44: Concatenate the noisy latent variables along the frame number dimension to obtain a latent representation adapted to video generation.

[0030] Furthermore, in step S7, the main feature q l The calculation formula is as follows:

[0031] q l =q·mask;

[0032] The formula for calculating attention output is as follows:

[0033]

[0034] Among them, atten l d is the attention output that aligns the text features of the l-th subject with the subject features, and d is the q l and k l Dimensions.

[0035] Furthermore, in step S9, the formula for obtaining the latent representation is as follows:

[0036]

[0037] Where ω is the weight, ε is the U-Net network, and ε(z) t ,p) are conditional latent variables, ε(z) t ) is an unconditional latent variable.

[0038] The present invention has the following advantages over the prior art:

[0039] (1) Distinguish between different subjects

[0040] Subject perception and attention decoupling separate different subjects while aligning only the information of one subject at a time during attention calculation, masking the information of other subjects. Attention calculation requires aligning the frame features and text features of each subject. Our decoupling attention extracts each subject and aligns it with its corresponding text features, while excluding other objects from the calculation. This ensures that each subject only exhibits actions corresponding to its own text in the final result.

[0041] (2) Improved flexibility and efficiency

[0042] This invention is built upon a training-free architecture. Existing image-based video processing can generate text-based motion animations when dealing with a single subject, but when multiple subjects are involved, it fails to distinguish between them, easily leading to confusion in appearance and action, resulting in missing subjects and chaotic actions. Training-free methods, built upon these single-subject models, avoid the cumbersome training process, significantly saving time. They are typically designed for seamless integration with existing models or systems, offering plug-and-play functionality. This provides users with great convenience. It also yields excellent results when dealing with missing multi-subject data. Attached Figure Description

[0043] Figure 1 This is a flowchart illustrating the graph-generated video method based on training-free strategy and multi-agent attention alignment in an embodiment of the present invention.

[0044] Figure 2 This is a schematic diagram of the training-free sampling framework structure in an embodiment of the present invention;

[0045] Figure 3 These are some of the experimental results in the embodiments of the present invention, where (a) is experimental result one and (b) is experimental result two. Detailed Implementation

[0046] The embodiments of the present invention are described in detail below. These embodiments are implemented based on the technical solution of the present invention, and provide detailed implementation methods and specific operation processes. However, the scope of protection of the present invention is not limited to the following embodiments.

[0047] Example 1

[0048] The task objectives for generating video from multiple subjects are explained below:

[0049] Image-to-video generation aims to transform static images into dynamic animations guided by specific input conditions. While there are various guiding conditions for image-to-video generation, such as trajectory, pose, optical flow, and text, most of these methods focus on a single subject, whereas multi-subject scenarios are common in reality. In this invention, we propose a training-free, decoupled diffusion model for multi-subject image-to-video generation. Specifically, we use subject perception to process the input text and images, matching different texts with their corresponding subjects to obtain a series of individual texts and their corresponding subjects. After decoupling the attention of each subject, the matched text and subject are bound together, the remaining subjects are masked, and finally, the generated attention is merged to replace the original attention. This ensures that each subject focuses only on its own region while ignoring the regions of other subjects, effectively solving the problem of overlapping appearances and movements between subjects in multi-subject scenarios, guaranteeing independent movement of each subject, and achieving high-quality animation effects.

[0050] We will achieve the above-mentioned task objectives through the following steps:

[0051] 1. Subject Awareness: To align different subjects with their respective text descriptions and avoid conflicts with other subjects in subsequent attention calculations, we use subject awareness. After inputting text and an image, we use a visual awareness method to segment the input image to obtain masks for different subjects in the image. These masks are further processed to preserve their respective parts while masking other subjects, meaning that the regions of other subjects are set to 0, and the rest to 1. To extract the region of each subject more accurately, we use a text awareness method to divide the complete text cue into multiple descriptions, each containing only one subject, and store them in a set. The cue in the set corresponds one-to-one with the subjects in the input image. Each cue in the set serves as a segmentation guide, sequentially segmenting the subjects in the image that match the text cue and accurately obtaining the mask for each subject. Then, the mask list obtained by inverting and masking other subjects is further refined. Finally, each mask only reveals a single subject, masking other subjects, and aligns each subject with the corresponding text during attention calculation.

[0052] 2. Attention Decoupling: In existing models, the semantics of multiple entities can become entangled during attention processing, leading to feature mixing. This confusion results in inconsistent entity appearances and actions that do not match the semantic descriptions. To address this issue, we decouple different attentions by redesigning the alignment between each entity's image features and the corresponding text embeddings in the frame image during U-Net processing. We use the Mask-based Query Generation (MQG) method to decouple the features of different entities and align these features with their corresponding text features. The decoupled attentions for each entity are then recombined into a single attention consistent with the U-Net network parameters. We call this combined computation of MQG and attention fusion "Entity-Aware Attention."

[0053] Example 2

[0054] This invention proposes a graph-to-video generation method that combines a training-free strategy with multi-subject attention alignment. By employing subject perception and attention decoupling, it achieves multi-subject animation generation, further enriching the expressiveness of graph-to-video tasks. Specifically, subject perception is used to process the input text and images, accurately matching different text prompts with their corresponding subjects to generate a series of independent text descriptions combined with their respective subjects. Subsequently, after decoupling the subject attention of the original model, the matched text and subjects are bound together, while other non-target subjects are masked. Finally, the generated independent attention features are recombinated to replace the attention mechanism in the original model. This method ensures that each subject focuses on its own region, ignoring interference from other subjects, effectively avoiding the problem of overlapping actions between subjects in multi-subject scenes, and achieving independent movement of each subject and high-quality animation generation.

[0055] Furthermore, this invention uses image segmentation, text description segmentation, subject alignment operations, and attention re-fusion operations to align the various subjects and text descriptions in the image together. The aim is to mask other objects while allowing all subjects to move as much as possible according to the matched text semantics. Its core steps are as follows:

[0056] Step 1: Obtain the input text and image.

[0057] Step 2: Using text segmentation technology, the description of each subject is extracted independently, and a mask for each subject is generated during image segmentation using these independent texts. To reduce the influence of other subjects, each mask is processed as follows: the pixel values ​​of the region corresponding to the subject in the mask are set to 1, and the pixel values ​​of the regions of other subjects are set to 0. Simultaneously, the region representing the subject in the mask is again set to 1 to ensure that the displayed subject has greater freedom of movement and less interference during its motion.

[0058] Step 3: Input the extracted mask, separated text, and corresponding image into a pre-trained image-to-video (I2V) model based on a diffusion model. This allows the model to combine text descriptions and mask information to generate video results corresponding to the input content.

[0059] Step 4: After processing the input image with a pre-trained encoder to obtain latent variables and adding noise, copy the same number of frames as the generated video, add a frame-number dimension to the noise, and then stitch all the noise together in the frame-number dimension.

[0060] Step 5: Use the pre-trained clip text encoder to extract features from each subject and copy several video frames, storing them sequentially in the list set p.

[0061] Step 6: Input the obtained set p, noise, and mask into the U-Net network.

[0062] Step 7: When calculating attention in U-Net, use a mask to obtain the features of each subject, and the subject features q l Matching text features k l v l First, calculate q using the dot product. l With k l The similarity is used to obtain an attention score (weight), which represents the relevance of each input to other inputs. These weights are then normalized (usually using softmax) to ensure that the weights sum to 1. Finally, these normalized weights are applied to the corresponding value vectors to generate the attention output.

[0063] Step 8: The multiple attentions obtained above are simply concatenated into a single attention that matches the pre-trained model, replacing the original attentions. This process then yields a denoised latent variable.

[0064] Step 9: The latent variables for one-step denoising are used to generate the final hidden representation using a classifier-free guidance method.

[0065] Step 10: Repeat the noise reduction process T times to obtain a clean animation video.

[0066] Example 3

[0067] like Figure 1 As shown, this embodiment further illustrates the method in Embodiment 2, and its specific steps are as follows:

[0068] Step 1: Obtain the input text (text) and image (x0) (human input).

[0069] Step 2: Using the text separation method (TP), the descriptive text containing multiple subjects is separated, and the description of each subject is stored in a separate set. Next, using these separated text descriptions, the visual segmentation (VP) method is employed to perform more precise segmentation of each subject in the image, generating corresponding subject region masks. Then, perception processing is performed on each mask: in each mask, the pixel value of the corresponding subject region is set to 1, and the pixel values ​​of other subject regions are set to 0, ensuring that each mask accurately represents the region of its corresponding subject. The subject region masks are obtained as follows:

[0070] masks=perception( VP(x0,TP(test)) ) (1)

[0071] Step 3: Input the extracted mask, separated text, and corresponding image into a pre-trained image-to-video (I2V) model based on a diffusion model. This allows the model to combine text descriptions and mask information to generate video results corresponding to the input content.

[0072] Step 4: After converting the input image into latent variables using a pre-trained encoder, noise is added to them. Next, this latent variable is copied multiple times, ensuring the number matches the frame rate of the target generated video. Then, noise is added to each copy of the latent variable along the frame rate dimension. Finally, these noisy latent variables are concatenated along the frame rate dimension to obtain a latent representation adapted for video generation. The specific processing method is as follows:

[0073] z 0 =E(x0) (2)

[0074]

[0075] Where E is the pre-trained encoder, z 0 This indicates that the input image encoding will be used as the first frame. It is the latent variable from frame 0 to frame f-1 at time t.

[0076] Step 5: Use the CLIP text encoder to extract features for each subject, and copy each feature to the same number of video frames. Then, store these copied features in a list set p for later use.

[0077] p = clip(TP(text)) (4)

[0078] Step 6: Combine the obtained set p and noise. The masks are fed into the U-Net network.

[0079] Step 7: When calculating cross-attention in the U-Net network, first use a mask to extract the feature q of each subject from the original multiple subject features q. l Main feature q l Matching text features k l and v l Perform dot product operation (k) l and v l (Obtained from set p), calculate q. l With k lThe similarity between features is used to obtain attention scores (weights), representing the relevance of each subject feature to other inputs. Next, these attention weights are normalized using softmax to ensure the sum of the weights is 1. Finally, the normalized attention weights are applied to the corresponding value vector v to generate the final attention output. The specific processing steps are as follows:

[0080] q l =q·mask (5)

[0081]

[0082] Among them, atten l It is the cross-attention feature output of the l-th subject.

[0083] Step 8: The multiple attentions obtained above are concatenated and normalized with the sum of all masks to obtain an attention deattention that matches the pre-trained image-to-video (I2V) model. This deattention replaces the original attention module in the U-Net network, resulting in a denoising step with the following latent variables:

[0084]

[0085] Among them, mask l It is the mask of the l-th subject, and n is the number of subjects.

[0086] Step 9: Generate the final latent variables using a classifier-free guidance method through a one-step denoising process. This process combines unconditional and conditional latent variable prediction to guide the generation of a latent representation that better matches the input conditions. The final latent representation, obtained after noise removal, is a compact expression of the generated content. The latent representation is obtained as follows:

[0087]

[0088] Where ω is the weight, ε is the U-Net network, and ε(z) t ,p) are conditional latent variables, ε(z) t ) is an unconditional latent variable.

[0089] Step 10: By repeating the denoising process T times, noise is gradually eliminated, ultimately generating a clear animated video. In each iteration, noise is gradually reduced, and video frames become clearer until a complete denoising result is obtained, thus generating a smooth and clean animated video.

[0090] This invention effectively solves the problem of motion confusion between subjects in multi-subject generated animation by introducing a subject perception and attention decoupling mechanism. Existing I2V technology often fails to effectively distinguish between different subjects when processing multiple subjects, resulting in intertwined actions and features in the generated animation, leading to inconsistent appearances or mismatched actions. To achieve clear subject differentiation, this invention independently segments each subject in the image during the input stage, generates a corresponding mask, and matches it one-to-one with the text description. In this way, each subject only exhibits actions consistent with its text description in the animation. This method significantly improves the accuracy and coherence of the generated animation, allowing viewers to more intuitively understand the behavior and features of each subject, enhancing the expressiveness and entertainment value of the animation.

[0091] This invention employs a training-free architecture, significantly simplifying the complexity of model construction and integration. Existing I2V models typically require training on large and diverse multi-agent datasets, which are relatively scarce in practical applications, limiting the model's applicability and generalization ability. Furthermore, traditional model training is not only time-consuming but also consumes substantial computational resources, posing a significant burden for many users. In contrast, this invention can be quickly integrated into existing systems without cumbersome training. This means users only need to provide simple text and image inputs to generate animations for multi-agent scenes, greatly lowering the technical barrier and enhancing application flexibility. This design not only makes it more convenient for users but also effectively improves the model's application efficiency and practical value.

[0092] This invention combines text and mask information to achieve independent and diverse motion representations of the subject, thus ensuring the richness and accuracy of the animation content. Existing I2V technology is often limited by model structure and training data when generating motion, resulting in often monotonous, mechanical movements lacking vividness and creativity. Furthermore, traditional models typically require users to make complex adjustments to generation parameters, making the process cumbersome and providing a poor user experience. To address these issues, this invention not only achieves independent control of each subject through subject perception but also utilizes an attention decoupling mechanism, allowing each subject to move freely according to its own textual descriptions, showcasing richer emotions and dynamic changes. Users can easily create high-quality animation content without needing to delve into the complexity of the model's internal parameters. This plug-and-play feature greatly improves user creation efficiency and satisfaction, allowing users to focus on the creation process itself rather than being bogged down in technical details.

[0093] Example 4

[0094] Existing image-to-video (I2V) generation methods are primarily based on diffusion models, attracting significant attention due to their advantages in generation controllability and high fidelity. With ongoing exploration of motion control, modulation modules, and spatiotemporal attention mechanisms, these methods have gradually improved I2V generation techniques and performed well on single-subject images. However, when dealing with multi-subject scenes, these methods often struggle to effectively distinguish between different subjects. Their image and text components interact in a coupled manner, potentially leading to visual confusion in the generated video, where the appearance and actions of subjects often do not match the text cues. In reality, multi-subject images are far more common than single-subject images in real-world scenarios, and the shortcomings of existing methods in this regard severely limit the application of I2V technology.

[0095] To address this limitation, this invention proposes a training-free diffusion method for multi-agent I2V generation. Figure 2 This invention demonstrates a training-free sampling framework. A pre-trained video model's VAE encodes video frames, composed of images and noise, into a latent space, generating latent variables. After the denoising process, the network converts the generated video back to pixel space. The framework comprises two parts: a pre-trained U-Net network and a subject perception and attention decomposition part. The U-Net network aligns the received latent variables and text features and performs iterative denoising. Before sampling, subject perception processes the text description and image, identifying and matching each subject with its corresponding text description. During sampling, our attention decomposition processes the video frames, decomposing different subjects, binding them to their respective text descriptions, and masking other irrelevant subjects. Subject perception combines visual and textual information, while attention decomposition replaces the original attention with subject-perception attention, binding each subject in the video to its corresponding text cue. In this way, the actions of each subject are accurately aligned with its text description and do not interfere with each other during generation.

[0096] The attention decoupling part of this invention will be further described in detail below, explaining how different subjects focus only on themselves and perform independent actions. First, in the original model, latent variables and text cues are input into the U-Net network to generate the original query from video frames featuring the input image. Next, different subjects in the original query are masked to obtain a query containing only the features of a single subject, which is stored in the q set. Simultaneously, the separated text is sequentially fed into CLIP to extract the text features corresponding to each subject. Based on different text inputs, feature sets k and v are obtained. Since the separated text is used to segment the image and a mask is generated through subject perception, q obtained from the mask segmentation naturally corresponds to k and v, requiring no additional matching. Attention fusion performs cross-attention calculations on q, k, and v to generate attention values ​​for multiple subjects. Finally, these attention values ​​are summed and normalized together with the merged mask to generate a decoupled single attention map (DeAttention). Some experimental results of this invention are shown below. Figure 3 As shown in (a) and (b) in the figure.

[0097] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A graph-based video generation method based on training-free strategy and multi-agent attention alignment, characterized in that, Includes the following steps: S1: Get the input text and image; S2: Separate the input text and segment the input image to obtain the mask of each subject, and perform perceptual processing on the mask; S3: Input the extracted mask, the separated text, and the corresponding input image into the pre-trained image-to-video (I2V) model based on the diffusion model; S4: Encode the image, then copy it multiple times, and add noise to create additional noise; S5: Feed the separated text into the CLIP text encoder to extract features for each subject, and copy each subject feature to the same number of frames as the target generated video. Store these copied subject features in a list set. middle; S6: Combine the sets obtained above Masks and noise for each main region are input into the U-Net network; S7: When computing cross-attention in the U-Net network, use a mask from multiple subject features. The characteristics of each subject are obtained in the process. Main features Matching text features and First calculate using the dot product. and The similarity between inputs is used to obtain an attention weight, representing the relevance of each input to other inputs. Then, the attention weights are normalized, and finally, these normalized attention weights are applied to the corresponding value vectors to generate the attention output. and From the set Obtain from, Indicates the first One entity; S8: Concatenate multiple attention outputs and normalize the sum of all masks to obtain an attention DeAttention that matches the pre-trained image-to-video (I2V) model based on the diffusion model. Replace the attention in the original attention module in the U-Net network with this DeAttention ... S9: Using a one-step denoising method, the classifier-free guidance method is used to generate the final latent variables and obtain the latent representation; S10: By repeating the denoising process T times, noise is gradually eliminated to obtain the target generated video; In step S2, the specific processing procedure is as follows: S21: By separating the input text, the descriptive text containing multiple subjects is separated, and the description of each subject is stored in a separate set; S22: Using the separated text description, segment each subject in the image through visual segmentation processing to generate the corresponding subject region mask; S23: Then perform perception processing on each mask: In each mask, set the pixel value of the corresponding subject area to 1, and set the pixel value of other subjects area to 0, so that each mask accurately represents the area of ​​its corresponding subject.

2. The graph-based video generation method based on training-free strategy and multi-agent attention alignment according to claim 1, characterized in that, In step S4, the specific processing procedure is as follows: S41: After the input image is transformed into latent variables by a pre-trained encoder, it is then subjected to noise processing. S42: Copy the latent variables multiple times, making the number of copies the same as the number of frames in the target generated video; S43: Add noise in one frame dimension to each latent variable; S44: Concatenate the noisy latent variables along the frame number dimension to obtain a latent representation adapted to video generation.

3. The graph-based video generation method based on training-free strategy and multi-agent attention alignment according to claim 2, characterized in that, In step S7, the main features The calculation formula is as follows: ； The formula for calculating attention output is as follows: ； in, It is the first Attention output that aligns the textual features of each subject with the subject's features. yes and Dimensions.

4. The graph-based video generation method based on training-free strategy and multi-agent attention alignment according to claim 3, characterized in that, In step S9, the formula for obtaining the latent representation is as follows: ； in, It's weight. For U-Net networks, It is a conditional latent variable. It is an unconditional latent variable.

Images