Long video character scene decoupling generation method and system based on multimodal diffusion

Abstract

A method and system for decoupling and generating long videos based on multimodal diffusion of characters and scenes relates to the field of image processing technology, including: S1, synthesizing high-level features of characters and scenes through a SigLIP encoder and a DINOv2 encoder; S2, cross-modal feature fusion of the high-level features to obtain joint features and compressing them to obtain compact vectors; S3, generating text features according to text prompts; S4, generating latent codes through a causal 3D convolutional encoder for the input video, and reducing the dimension by splicing it with the memory state after passing through a linear projection matrix to obtain a segmented latent vector sequence; S5, performing decoupled perception generation on the segmented latent vector sequence through an improved 3D-UNet, and then reconstructing an RGB video segment sequence through deconvolution upsampling after deterministic sampling; the present invention solves the key problems of rough dynamic control, limited generation length and excessive resource consumption in long video generation, and significantly improves the quality and efficiency of generated videos.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method and system for generating character-scene decoupling in long videos based on multimodal diffusion. Background Art

[0002] Traditional video generation techniques typically optimize the character and scene as a whole end-to-end, preventing users from independently adjusting character movements or scene dynamics. For example, modifying a character's running speed could disrupt the natural rhythm of swaying leaves in the background. Furthermore, existing methods rely on global attention mechanisms to model long-term temporal relationships. However, as video length increases, the model struggles to maintain physical consistency across segments (e.g., character motion trajectory interruptions and sudden changes in lighting). Furthermore, the computational cost of generating full-frame, high-resolution images increases exponentially, limiting practical applications.

[0003] Existing long video generation methods have significant limitations in terms of independent control of character and scene dynamics, ensuring temporal consistency, and high-resolution generation efficiency. For example, traditional end-to-end models cause interference between character movements and background dynamics due to coupled optimization. Pure attention mechanisms are unable to maintain the continuous motion logic of minute-level videos, and full-frame high-resolution generation faces a huge computational load.

[0004] To address these issues, this solution aims to build a long video generation framework that combines flexible control capabilities with efficient generation efficiency. By decoupling the character and scene generation paths, the two can be independently optimized based on conditions such as text and identity, avoiding mutual interference. Causal timing constraints and memory recursion mechanisms are designed to ensure the dynamic coherence of minute-long videos. Furthermore, a divide-and-conquer strategy is employed to differentiate between characters and backgrounds—super-resolution enhancement is performed on visually focused areas (such as people), while lightweight interpolation is used for non-focus areas (such as the sky), achieving film-quality resolution output with limited computing power. This design is motivated by the practical needs of storyboard editing and virtual production in the film and television industrialization process, aiming to provide content creators with a long video generation tool that combines high-precision control with low resource consumption. Summary of the Invention

[0005] In order to overcome the deficiencies in the background technology, the present invention discloses a method and system for generating character-scene decoupling in long videos based on multimodal diffusion.

[0006] In order to achieve the above-mentioned object of the invention, the present invention adopts the following technical solutions:

[0007] First, a method for decoupling and generating character scenes in long videos based on multimodal diffusion includes the following steps:

[0008] S1. Extract high-level semantic features and fine-grained texture features of the input image through the SigLIP encoder and the DINOv2 encoder, and extract high-level semantic features and fine-grained texture features of the input video through the SigLIP encoder and the DINOv2 encoder, and synthesize the high-level semantic features and fine-grained texture features in proportion.

[0009] S2. Perform cross-modal feature fusion on high-level semantic features and fine-grained texture features to obtain joint features, and compress the joint features using the Q-Former projection head through the character branch to obtain a compact character identity vector, and compress the joint features using the Q-Former projection head through the scene branch to obtain a compact scene semantic vector;

[0010] S3, text prompt P generates text features through T5-XL model encoding;

[0011] S4: After temporal segmentation of the input video, a latent code is generated through a causal 3D convolutional encoder. The character identity compact vector, scene semantic compact vector, and text features are embedded into the latent space through a linear projection matrix and fused with the latent code to obtain an enhanced latent code. The enhanced latent code is then concatenated with the memory state updated by the attention mechanism and the dimensionality is reduced to obtain a segmented latent vector sequence, and the global memory state is output.

[0012] S5. The noise latent vector sequence is injected into the segmented latent vector, and the noise latent vector is generated by the improved 3D-UNet for decoupled perception. The DDIM deterministic sampling strategy is then used to iteratively denoise the noise latent vector to obtain the final sampled output. Finally, the final sampled output is input into the symmetric 3D convolution decoder, and the RGB video segment sequence is reconstructed by deconvolution upsampling.

[0013] S6. The RGB video segment sequence and the global memory state are enhanced by segment alignment and temporal interpolation to synthesize the final long video.

[0014] Specifically, step S1 includes the following steps:

[0015] S11, input image , and perform scaling normalization to obtain the normalized image ; Where H is the image height, W is the image width, the unit is pixel; 3 is the RGB three channels;

[0016] S12, input video Get a set of key frames by uniform sampling V K ={ V 1 , V 2 ,..., Vk}, and for each keyframe V k Perform scaling normalization to obtain , thus obtaining a normalized key frame set ; T is the number of video frames;

[0017] S13. Extract normalized image through SigLIP encoder High-level semantic features of images , , and extract the normalized image through the DINOv2 encoder Fine-grained texture features of images , ;

[0018] S14. Extract normalized key frame set through SigLIP encoder The high-level semantic features of each frame are pooled and the high-level semantic features of the video are obtained by temporal average pooling. , and extract the normalized keyframe set through the DINOv2 encoder The fine-grained texture features of each frame of the image are pooled and the video fine-grained texture features are obtained by temporal average pooling. ;

[0019] S15, high-level semantic features of the image High-level semantic features of videos Proportionally synthesize high-level semantic features , and the fine-grained texture features of the image Fine-grained texture features of videos Proportional synthesis of fine-grained texture features .

[0020] Specifically, step S2 includes the following steps:

[0021] S21, high-level semantic features Broadcasting to spatial dimensions and fine-grained texture features Splicing along the channel dimension and performing cross-modal optimization through two layers of Transformer to obtain joint features , ;

[0022] S22, the learnable query vector of the role branch is combined with the joint feature through the cross attention layer After interaction, the self-attention Transformer block integrates global information and linear projection, and then uses the Q-Former projection head to compress the character identity compact vector ;

[0023] S23, the learnable query vector of the scene branch is combined with the joint feature through the cross attention layer After interaction, the self-attention Transformer block integrates global information and linear projection, and then uses the Q-Former projection head to compress the scene semantics compact vector .

[0024] Specifically, step S4 includes the following steps:

[0025] S41, input video Cut into N video segments of fixed length L , each video segment The latent code is generated by processing it through a causal 3D convolutional encoder z i ;

[0026] S42, role identity compact vector , scene semantic compact vector and text features Embedded into the latent space through the linear projection matrix and compared with the latent code z i Fusion Enhanced Latent Code ;

[0027] S43, the memory state corresponding to the i-1th video segment M i-1 and enhanced latent coding The memory state corresponding to the i-th video segment is updated segment by segment through the attention mechanism M i , thus obtaining the global memory state M N , M N ={ M 1 , M 2 ,..., M i}, where i=1,2,...,N;

[0028] S44, enhance the potential coding corresponding to the i-th video segment The corresponding memory status M i After flattening and concatenation, the segmented latent vector is output through linear dimensionality reduction , and obtain the segmented latent vector sequence .

[0029] Specifically, step S5 includes the following steps:

[0030] S51, first segment the potential vector sequence Noise injection: superimpose time-varying Gaussian noise with linearly increasing time steps t , generating a noise latent vector , ;in is the pollution noise scheduling coefficient at time step t; and according to the role identity compact vector , scene semantic compact vector and text features Synchronously initialize the three path conditions to generate identity key-value pairs , scene key-value pairs and conditional embedding features ;

[0031] S52, Noise Latent Vector Input improved 3D-UNet according to identity key-value pairs , scene key-value pairs and conditional embedding features Perform decoupling perception to generate character path features and scene path characteristics ;

[0032] S53, the global memory state is extended to the spatial dimension after projection , and the skip connection feature of 3D-UNet Splicing as memory splicing feature , and finally through the convolution kernel Implement fusion to generate memory fusion features ,in is the convolution function; The weight of ;

[0033] S54. Role Path Characteristics , scene path features and memory fusion features Generating prediction noise of 3D-UNet through feature synthesis layer : ,in is the GeLU activation function, is the hidden layer projection matrix, is the output layer weight;

[0034] S55, based on the predicted noise DDIM sampling is used to sample the noise latent vector Perform iterative denoising to obtain the final sample output : ,in is the denoising scheduling coefficient at time step t; is the denoising scheduling coefficient after one iteration of time step t; is the noise latent vector The tentative vector after one step of repair;

[0035] S56, output the final sample Input symmetric 3D convolution decoder, reconstructed into RGB video segment through deconvolution upsampling , get the RGB video segment sequence .

[0036] Specifically, step S6 includes the following steps:

[0037] S61, first extract motion features, for adjacent video segments and , first extract video segments through pre-trained I3D network The last two frames Movement characteristics and extract video segments The first two frames Movement characteristics ; Then calculate the alignment loss , , and then according to the alignment loss Perform boundary optimization to obtain the final output aligned segment sequence ;in, Represents L2 norm operation; Represents L1 norm operation; represents the time gradient operator; ;I3D() represents the pre-trained I3D network;

[0038] S62, in the alignment segment The motion-aware neural network interpolation algorithm is used to evenly insert 4 intermediate frames and output an enhanced segmented sequence. ;

[0039] S63, strengthen the segment sequence Constructing a complete video stream through time-series stitching , and the complete video stream Perform global chroma correction and output the final long video with consistent color .

[0040] Specifically, the overlapping area of ​​adjacent enhanced segments uses the Hamming window function Perform weighted average fusion Overwrite the complete video stream The corresponding position.

[0041] Specifically, the scaling normalization process in step S11 is implemented based on the following formula:

[0042] ;

[0043] Among them, resize() represents a scaling function.

[0044] Specifically, step S22 is as follows:

[0045] The character branch initializes 16 learnable query vectors , through the cross attention layer and the joint feature interaction, and then integrate the global information through 4 layers of self-attention Transformer blocks, and linearly project it into dimensional role identity vector , and finally use the Q-Former projection head to transform the character identity vector Compressed into a compact character identity vector .

[0046] Specifically, step S41 is as follows:

[0047] The causal 3D convolutional encoder consists of 3 residual blocks, 3 convolutional layers, and 2 pooling layers; the input is a video segment , the output is the potential code corresponding to the video segment z i ;

[0048] Input video segment First, the causal 3D convolutional encoder is used for layer-by-layer processing: the first convolutional layer uses a kernel size of The convolution of the network outputs a 64-channel feature map. The time axis is padded with 1 frame on the left to ensure causality, and the space is padded with 3 pixels to maintain resolution. The activation function is GELU to obtain the intermediate feature. ; The second layer passes through the kernel size , convolution with 128 channels, time step 2, downsampling with spatial step 2 to a resolution of 128×128, and applying a temporal causal mask, with an output dimension of ; The third layer uses a kernel size of , convolution with 256 channels, further downsampled to 64×64, to obtain the feature map ;

[0049] Subsequently, the features are optimized through three residual blocks ResBlock: each residual block contains GroupNorm, GELU activation, causal 3D convolution and skip connection to keep the feature dimension unchanged; average pooling is applied to the processed features along the time axis to compress the time step from 8 to 4, and the feature map is obtained. Finally, the number of channels is maintained through 1×1×1 convolution, and the resolution is reduced to 32×32 in combination with spatial maximum pooling to generate the final low-resolution potential code .

[0050] On the other hand, the present invention also discloses a long video character scene decoupling generation system based on multimodal diffusion, which is based on the long video character scene decoupling generation method based on multimodal diffusion described in the first aspect and includes the following modules:

[0051] An advanced feature extraction module is used to extract high-level semantic features and fine-grained texture features of the input image through the SigLIP encoder and the DINOv2 encoder, and to extract high-level semantic features and fine-grained texture features of the input video through the SigLIP encoder and the DINOv2 encoder, and to synthesize the high-level semantic features and fine-grained texture features in proportion;

[0052] The feature fusion and compression module is used to perform cross-modal feature fusion on high-level semantic features and fine-grained texture features to obtain joint features, and compress the joint features using the Q-Former projection head through the character branch to obtain a compact character identity vector, and compress the joint features using the Q-Former projection head through the scene branch to obtain a compact scene semantic vector;

[0053] The text feature extraction module is used to encode the text prompt P through the T5-XL model to generate text features;

[0054] The block-wise latent diffusion module is used to segment the input video into time series and generate latent codes through a causal 3D convolutional encoder. The character identity compact vector, scene semantic compact vector, and text features are embedded into the latent space through a linear projection matrix and fused with the latent code to obtain an enhanced latent code. The enhanced latent code is then concatenated with the memory state updated by the attention mechanism and the dimensionality is reduced to obtain a segmented latent vector sequence, which then outputs the global memory state.

[0055] The decoupled reconstruction video module is used to inject noise into the segmented latent vector sequence to generate a noisy latent vector. The noise latent vector is then decoupled and generated through an improved 3D-UNet. The DDIM deterministic sampling strategy is then used to iteratively denoise the noise latent vector to obtain the final sampled output. The final sampled output is input into a symmetric 3D convolutional decoder, and the RGB video segment sequence is reconstructed through deconvolution upsampling.

[0056] The block autoregressive generation module is used to synthesize the final long video by strengthening the RGB video segment sequence and the global memory state through segment alignment and temporal interpolation.

[0057] The present invention proposes a long video character scene decoupling generation method based on multimodal diffusion. Its core is to solve the key problems of rough dynamic control, limited generation length and excessive resource consumption in long video generation through multimodal condition decoupling, causal spatiotemporal modeling and branch synthesis technology, thereby significantly improving the quality and efficiency of generated videos.

[0058] Furthermore, by integrating multimodal conditions with the diffusion model, the present invention achieves:

[0059] 1) High-definition image quality: Progressive optimization of the diffusion model delivers detail fidelity far exceeding traditional methods;

[0060] 2) Refined independent control: Action / environment phrases dynamically decouple and regulate characters and scenes through a diffusion process;

[0061] 3) Long time series are more stable: Diffusion generation combined with a memory mechanism significantly reduces interruptions and flickers in long videos;

[0062] Furthermore, the present invention designs a diffusion generation architecture that independently optimizes characters and scenes, uses text-driven multimodal conditions to separate and control character behavior and background dynamics; introduces causal 3D convolution and a global memory mechanism to maintain cross-block temporal consistency; the present invention can support the demand for efficient and controllable long video generation in scenarios such as film and television pre-visualization and virtual character driving, breaking through the bottleneck of existing methods in decoupling editing capabilities and generation length.

[0063] Furthermore, the present invention provides a long video generation tool with high-precision control and low resource consumption for content creators who have needs for storyboard editing and virtual production in the film and television industrialization process. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0065] Figure 1 Flowchart of a method for decoupling and generating long video characters and scenes based on multimodal diffusion according to an embodiment of the present invention;

[0066] Figure 2 2 is a schematic structural diagram of a long video character scene decoupling generation system based on multimodal diffusion according to an embodiment of the present invention;

[0067] Figure 3 3 is a structural diagram of a long video character scene decoupling generation device based on multimodal diffusion provided according to an embodiment of the present invention. DETAILED DESCRIPTION

[0068] The present invention can be explained in detail through the following embodiments. The purpose of disclosing the present invention is to protect all technical improvements within the scope of the present invention. In the description of the present invention, it should be understood that if there are terms such as "upper", "lower", "front", "back", "left", "right", etc. to indicate directions or positional relationships, they only correspond to the drawings of this application for the convenience of describing the present invention, and do not indicate or imply that the device or element referred to must have a specific direction.

[0069] Example 1

[0070] refer to Figure 1 According to this embodiment, a method for generating character-scene decoupling in long videos based on multimodal diffusion is disclosed, which includes the following steps:

[0071] The input includes the input image , where H is the image height, W is the image width, and the unit is pixel; 3 is the RGB three-channel, and the value range of RGB is 0-255; text prompt P; input video , T is the number of video frames.

[0072] S1. Extract high-level semantic features and fine-grained texture features of the input image through the SigLIP encoder and the DINOv2 encoder, and extract high-level semantic features and fine-grained texture features of the input video through the SigLIP encoder and the DINOv2 encoder, and synthesize the high-level semantic features and fine-grained texture features in proportion.

[0073] Step S1 specifically includes the following steps:

[0074] Step 1: Image and video preprocessing:

[0075] S11, image preprocessing, input image , and perform scaling normalization to obtain the normalized image ; The formula for scaling normalization is ;

[0076] Among them, resize() represents the scaling function;

[0077] S12, video preprocessing, input video Get a set of key frames by uniform sampling V K ={ V 1 , V 2 ,..., V k}, and for each keyframe Vk Perform scaling normalization to obtain , thus obtaining a normalized key frame set ;

[0078] The uniform sampling is implemented using the following formula:

[0079] ;

[0080] Among them, uniform_sample( V ref , K ) is a uniform sampling function, K is the number of key frames; V k To represent the kth key frame, k={1,K}; V K is a set of key frames;

[0081] uniform_sample( V ref , K ) is a uniform sampling function, which means that K key frames are extracted from the input video at a fixed time interval as a key frame set. K is a positive integer. In this implementation, the default value is 8 and can be modified according to actual needs. The default fixed time interval is 0.5 seconds and can also be modified according to actual needs.

[0082] For each keyframe V k Perform scaling normalization to obtain The same scaling normalization formula as in step S11 is used, as follows:

[0083] ;

[0084] Normalized keyframe set Elements and keyframe collections in V K The elements correspond one to one in sequence.

[0085] Step 2: Two-stream visual feature extraction

[0086] S13. Extract normalized image through SigLIP encoder High-level semantic features of images , , and extract the normalized image through the DINOv2 encoder Fine-grained texture features of images , ;

[0087] Among them, SigLip() represents the SigLIP encoder; DINOv2() represents the DINOv2 encoder;

[0088] The SigLIP encoder, proposed by Google DeepMind, is an improved version of the CLIP (Contrastive Language-Image Pretraining) model. It focuses on multilingual vision-language encoding and integrates multiple techniques, such as caption-based pretraining, self-supervised losses (self-distillation, mask prediction), and online data filtering, into a unified training scheme. The SigLIP encoder achieves significant improvements in semantic understanding, localization, and dense feature extraction. It supports cross-lingual scenarios and performs well in both English and non-English tasks.

[0089] The DINOv2 encoder is a self-supervised visual foundation model developed by Meta AI. It belongs to the category of visual encoders in deep learning. Its core design goal is to learn universal visual feature representations from images without manual annotation and to demonstrate excellent performance in multiple downstream tasks such as classification, segmentation, and detection.

[0090] Extract normalized image through SigLIP encoder High-level semantic features of images Specifically include:

[0091] Normalized image First, it is divided into 24×24=576 non-overlapping 16×16 image blocks, each of which is flattened into a 768-dimensional vector and then passed through a learnable weight matrix Linear projection to 1024-dimensional embedding space to generate the initial embedding vector , the initial embedding vector and learnable positional encoding Add element by element to form the first initial marker sequence containing spatial position information , in the first initial tag sequence Header injection of learnable global CLS flags , forming the first complete input sequence ;

[0092] Learnable weight matrix Random initialization assignments (e.g. sampling from a normal distribution) are performed before model training, and their values ​​are automatically optimized during training through backpropagation;

[0093] Learnable positional encoding Similarly, the values ​​are randomly initialized and added element-by-element to the input vector during training, and their parameters are learned and updated together with the model.

[0094] In the first complete input sequence, the CLS tag The actual insertion is at the beginning of the sequence, and its display at the end is the visual representation of the fusion of the attention mechanism.

[0095] The first complete input sequence The input is a 12-layer Transformer encoder based on the ViT-B / 16 architecture. Each layer performs multi-head self-attention mechanism and MLP expansion processing in turn to obtain intermediate features, and optimizes the intermediate feature transfer through residual connection and layer normalization. Finally, the learned CLS tag is extracted from the 12th layer output sequence. , after learning CLS mark Mapped into high-level semantic feature vectors through the linear projection layer ,This vector represents the abstract semantics of the encoded image,such as role category, posture and relationship between objects;

[0096] Each layer sequentially performs the multi-head self-attention mechanism and MLP expansion processing as the core operations performed by each layer, specifically:

[0097] Each layer of Transformer encoder includes 8 attention heads, the dimension of the attention head is 128, and the calculation formula is , the multi-head self-attention mechanism is used to calculate the tag sequence With learnable global CLS markers Feature correlation, capturing long-range dependencies between image patches;

[0098] The MLP expansion ratio is 4, which means that the feature dimension is expanded from 1024 to 4096 and then restored to 1024, enhancing the nonlinear expression ability.

[0099] Multi-Head Attention is an extended form of the attention mechanism widely used in the Transformer model. It obtains the attention distribution of different subspaces of the input sequence by running multiple independent attention mechanisms in parallel, thereby more comprehensively capturing the potential multiple semantic associations in the sequence.

[0100] In multi-head attention, the first complete input sequence First, three different linear transformation layers are used to obtain the Query, Key, and Value vectors, corresponding to the formulas In the formula, Q, K, and V are represented by T, which represents the transpose function and softmax represents the normalized exponential function. These transformed vectors are divided into several "heads", each of which has its own independent query, key, and value matrices. For each head, a scaled dot-product attention operation is performed. Finally, the outputs of all heads are concatenated and then fused through a linear layer to obtain the final attention output vector. ;

[0101] Extract normalized image through DINOv2 encoder Fine-grained texture features of images Specifically include:

[0102] Normalize the image 4 pixels are cropped on each side to obtain a 376×376 effective area, which is divided into 14×14 blocks, resulting in a total of 676 non-overlapping blocks. Each non-overlapping block is flattened into a 14×14×3=588-dimensional vector, which is then transformed into a vector by the linear projection matrix. Mapped to a 768-dimensional embedding space and combined with a learnable positional encoding Add element by element to form the second initial tag sequence ;

[0103] Linear projection matrix By randomly initializing the assignment, its value is automatically optimized during the training process through back propagation.

[0104] Learnable positional encoding By randomly initializing the assignment and adding it to the input vector element by element during training, its parameters are learned and updated together with the model.

[0105] The second initial tag sequence Input the DINOv2 encoder based on the ViT-S / 14 architecture and extract the shallow feature map when forward propagating to the 3rd layer , the shallow feature map Reshape into a three-dimensional tensor of spatial dimension 26×26×768 to represent local detail responses (clothing details and background elements, such as clothing wrinkles, hair texture and background element edges). To enhance feature robustness and reduce computational load, apply the maximum pooling operation, set the kernel size to 2×2, and the stride to 2 for spatial downsampling, and output the downsampled feature map. , that is, the fine-grained texture features of the image.

[0106] Max Pooling is a spatial downsampling technique with a kernel size of 2×2 and a stride of 2, which means that the feature map in the shallow layer The kernel is traversed with a 2×2 sliding window, and the maximum value in each window is extracted as the output. The sliding step is 2 (i.e., it moves 2 pixels each time).

[0107] S14. Extract normalized key frame set through SigLIP encoder The high-level semantic features of each frame are pooled and the high-level semantic features of the video are obtained by temporal average pooling. , and extract the normalized keyframe set through the DINOv2 encoder The fine-grained texture features of each frame of the image are pooled and the video fine-grained texture features are obtained by temporal average pooling. ;

[0108] Iterate over the normalized keyframe set Each keyframe The corresponding image is then extracted through the SigLIP encoder to extract the image's high-level semantic features, and the temporal average pooling is performed to obtain the video's high-level semantic features. , corresponding formula ;

[0109] Iterate over the normalized keyframe set Each keyframe The corresponding image is then extracted through the DINOv2 encoder to extract the fine-grained texture features of the image, and the temporal average pooling is performed to obtain the fine-grained texture features of the video. , corresponding formula: ;

[0110] In the formula, SigLip() represents the SigLIP encoder; DINOv2() represents the DINOv2 encoder;

[0111] S15, high-level semantic features of the image High-level semantic features of videos Proportionally synthesize high-level semantic features , and the fine-grained texture features of the image Fine-grained texture features of videos Proportional synthesis of fine-grained texture features ;

[0112] S2. Perform cross-modal feature fusion on high-level semantic features and fine-grained texture features to obtain joint features, and compress the joint features using the Q-Former projection head through the character branch to obtain a compact character identity vector, and compress the joint features using the Q-Former projection head through the scene branch to obtain a compact scene semantic vector;

[0113] Step 3: Cross-modal feature fusion

[0114] S21, high-level semantic features Broadcast to spatial dimensions (dimension is 13×13) and fine-grained texture features Splicing along the channel dimension and performing cross-modal optimization through two layers of Transformer to obtain joint features , ;

[0115] Step 4: Role-Scene Decoupling

[0116] The role and scene decoupling phase uses two branches (role branch and scene branch) to process them independently:

[0117] S22, the learnable query vector of the role branch is combined with the joint feature through the cross attention layer After interaction, the self-attention Transformer block integrates global information and linear projection, and then uses the Q-Former projection head to compress the character identity compact vector ;

[0118] The cross-attention layer here is a separate attention layer, which is used to allow the query vector of the character branch to actively extract key information from the joint features;

[0119] The character branch initializes 16 learnable query vectors (feature extractor), through the cross attention layer and the joint feature The interaction is then integrated through a 4-layer self-attention Transformer block to form a character identity vector through linear projection. , and finally use the Q-Former projection head to transform the character identity vector Compressed into a compact character identity vector , the dimension is 512;

[0120] Specifically, the role identity vector (Optimization results for 16 query vectors) Input a dedicated Q-Former projection head for final compression, which consists of a cross-attention layer and a lightweight Transformer block: First, initialize 16 learnable target query vectors (Feature Distiller) as the attention query input, and and As key-value pairs, output interactive features through cross-attention calculation , , , then input two layers of Transformer blocks (each layer contains 4 self-attention heads and MLP expansion ratio is 4), and integrate cross-query information through residual connections to generate character context enhanced features (global features) , finally, through the character linear projection matrix Enhance the character context features corresponding to the target query vector Compressed into a 512-dimensional character identity compact vector ;in They are query projection matrix, key projection matrix and value projection matrix respectively. is the scaling factor, T represents the transpose function; softmax() represents the normalized exponential function, and CrossAttn() is the cross attention calculation function;

[0121] Character Linear Projection Matrix , query the projection matrix , key projection matrix and value projection matrix All are assigned by random initialization;

[0122] The character branch initializes 16 learnable query vectors Used to combine features from multiple modalities Extracting identity information from is the extracted intermediate features (role identity vector ) for compression and dimensionality reduction.

[0123] S23, the learnable query vector of the scene branch is combined with the joint feature through the cross attention layer After interaction, the self-attention Transformer block integrates global information and linear projection, and then uses the Q-Former projection head to compress the scene semantics compact vector .

[0124] The scene branches process independent query vectors in the same way , generating scene semantic compact vectors ;

[0125] The scene branch initializes 16 learnable query vectors (feature extractor), through the cross attention layer and the joint feature interaction, and then integrate the global information through 4 layers of self-attention Transformer blocks, and linearly project it into dimensional scene semantic vector Finally, the scene semantic vector is transformed using the Q-Former projection head Compressed into a 512-dimensional scene semantic compact vector , the dimension is 512;

[0126] Specifically, scene semantic vector (Optimization results for 16 query vectors) Input a dedicated Q-Former projection head for final compression, which consists of a cross-attention layer and a lightweight Transformer block: First, initialize 16 learnable target query vectors (Feature Distiller) as the attention query input, and and As key-value pairs, output interactive features through cross-attention calculation , , , then input two layers of Transformer blocks (each layer contains 4 self-attention heads and MLP expansion ratio is 4), and integrate cross-query information through residual connections to generate scene context enhanced features (global features) , finally, through the scene linear projection matrix Enhance the scene context features corresponding to the target query vector Compressed into a 512-dimensional character identity compact vector ;in They are query projection matrix, key projection matrix and value projection matrix respectively. is the scaling factor, T represents the transpose function; softmax() represents the normalized exponential function, and CrossAttn() is the cross attention calculation function;

[0127] Scene linear projection matrix , query the projection matrix , key projection matrix and value projection matrix All are assigned by random initialization;

[0128] The scene branch initializes 16 learnable query vectors For joint features from multimodal ) to extract scene information; and It is the extracted intermediate features (scene semantic vector ) for compression and dimensionality reduction.

[0129] The character branch focuses on entity details (such as character relationships), while the scene branch extracts overall semantics (such as events / environment). The two are complementary but heterogeneous. Decoupled learning in these two branches prevents interference between fine-grained character information and coarse-grained scene information, improving accuracy. Furthermore, dedicated branches can optimize cross-modal alignment.

[0130] The self-attention Transformer block is a neural network module built based on the self-attention mechanism. It is mainly used to process sequence data and can capture the dependencies between elements within the sequence. It is widely used in natural language processing, computer vision and other fields.

[0131] S3, text prompt P generates text features through T5-XL model encoding F text ;

[0132] Step 5: Text feature encoding

[0133] The text prompt P is processed by an independent T5-XL encoder to output 768-dimensional context-aware text features. , in is the number of text tokens, and T5_XL() is the T5-XL encoder.

[0134] The text prompt P generates text features through T5-XL model encoding F text Specifically include:

[0135] The text prompt P is processed by the SentencePiece tokenizer, which divides the natural language into subword units and generates a token sequence with a maximum length of 512. Each token is mapped to a 768-dimensional vector through the embedding layer and with absolute position encoding Element-wise addition generates position-aware input sequences , where the absolute position encoding Generated using sine / cosine functions;

[0136] Absolute position encoding Follow the sine-cosine function rule to preserve word order information, directly superimpose it on the word vector, and provide sequence order information for the model;

[0137] Embedding() is an embedding function. In machine learning, embedding() primarily refers to mapping discrete, high-dimensional data (such as text, images, and audio) into a low-dimensional, continuous vector space. This process generates a vector composed of real numbers that captures the underlying relationships and structure of the original data.

[0138] Position-aware input sequence Input T5-XL's 24-layer encoder for deep semantic modeling. Each layer of the encoder sequentially executes a multi-head self-attention mechanism, a feedforward neural network, residual connections, and layer normalization to finally output text features. .

[0139] The multi-head attention in this step includes 16 attention heads, and the dimension of the attention head is 48. By scaling the dot product formula Calculate feature correlation and implement global dependency modeling. The 16 attention heads are equivalent to 16 parallel processing "perspectives", and each head focuses on capturing the relationship between different subspaces (such as syntax, semantics, entity associations, etc.). The feedforward neural network includes two layers of linear transformation, each of which contains GeLU activation. The hidden layer dimension 768→4096→768 expands the semantic representation. The nonlinear expression ability is enhanced by dimensional expansion (768→4096). The GeLU activation function is smoother than ReLU, which improves training stability, and then shrinks back to 768. Finally, residual connection and layer normalization are performed to stabilize the gradient flow and effectively alleviate the gradient disappearance problem of deep networks.

[0140] Final output of steps S1-S3: compact character identity vector , scene semantic compact vector , text features .

[0141] S4. Block-wise latent diffusion initialization: After temporal segmentation of the input video, a latent code is generated through a causal 3D convolutional encoder. The character identity compact vector, scene semantic compact vector, and text features are embedded into the latent space through a linear projection matrix and fused with the latent code to obtain an enhanced latent code. This is then concatenated with the memory state updated by the attention mechanism and the dimensionality is reduced to obtain a segmented latent vector sequence, and the global memory state is output.

[0142] Input contains the original input video (Total number of frames is T, resolution is ) and the compact character identity vector , scene semantic compact vector .

[0143] S41, the original input video Cut into N video segments of fixed length L , each video segment The latent code is generated by processing it through a causal 3D convolutional encoder z i ;

[0144] First, the original input video Cut into N fixed-length video segments (each segment frames), and obtain a set of video segments (satisfying T=N×L), each video segment Processed by a causal 3D convolutional encoder to generate latent codes ,Causal convolution ensures that the encoding only depends on historical frames, avoiding future information leakage.

[0145] The causal 3D convolutional encoder consists of 3 convolutional layers, 3 residual blocks, and 2 pooling layers; the input is a video segment , the output is the potential code corresponding to the video segment z i ;

[0146] Input is the original input video The video segment collection obtained after cutting , where each video segment Contains 16 frames of raw RGB video (resolution 256×256);

[0147] Input video segment First, the causal 3D convolutional encoder is used for layer-by-layer processing: the first convolutional layer uses a kernel size of (time × height × width) convolution, output 64-channel feature map, the time axis is padded by 1 frame on the left to ensure causality (only historical frames participate in the current frame calculation), the space is padded by 3 pixels to maintain resolution, the activation function is GELU, and the intermediate feature is obtained ; The second layer passes through the kernel size , convolution with 128 channels, time step 2, downsampling with spatial step 2 to a resolution of 128×128, and applying a temporal causal mask, with an output dimension of ; The third layer uses a kernel size of , convolution with a channel number of 256, further downsampled to 64×64, to obtain the feature map ;

[0148] Subsequently, the features are optimized through three residual blocks ResBlock: each residual block contains GroupNorm (8 groups), GELU activation, causal 3D convolution (kernel size ) and jump connections to keep the feature dimension unchanged; apply average pooling (kernel size is 2×1×1, step size is 2) to the processed features along the time axis, compressing the time step from 8 to 4 to obtain the feature map Finally, the number of channels is maintained through 1×1×1 convolution, and the resolution is reduced to 32×32 in combination with spatial maximum pooling (kernel size 2×2) to generate the final low-resolution potential code .

[0149] The Causal 3D Convolutional Encoder is a neural network model that combines causal convolution, 3D convolution, and encoder architecture. It is designed specifically for processing spatiotemporal data (such as videos, medical images, and dynamic scenes). Its core goal is to extract spatiotemporal features while ensuring causality in the temporal dimension. This means that the output depends only on the current and past inputs. Its hierarchical downsampling structure gradually compresses the spatiotemporal dimensions (similar to a 3D U-Net), preventing future information leakage.

[0150] S42, role identity compact vector , scene semantic compact vector and text features Embedded into the latent space through the linear projection matrix and compared with the latent code z i Fusion Enhanced Latent Code ;

[0151] Next, the character identity compact vector Linear projection matrix via identity Character feature projection vector embedded into latent space ;

[0152] Scene Semantic Compact Vector Through the scene linear projection matrix Embedded into the latent space to obtain the scene feature projection vector ;

[0153] Text features Through the text linear projection matrix Embedded into the latent space to obtain the text feature projection vector ;

[0154] Then, enhanced latent codes are generated by channel dimension concatenation and 1×1×1 convolution fusion. , , Represents the 1×1 convolution kernel parameters (i.e., point convolution), which is used for cross-channel feature fusion.

[0155] Among them, Conv3D() is a three-dimensional convolution function, which is used for three-dimensional convolution fusion operations and is often used to process data with spatial and temporal dimensions (such as videos, medical images, spatiotemporal sequence data, etc.); concat() is a splicing function for channel dimension splicing; , , , All are randomly initialized and assigned values ​​for fine-tuning.

[0156] S43, the memory state corresponding to the i-1th video segmentM i-1 and enhanced latent coding The memory state corresponding to the i-th video segment is updated segment by segment through the attention mechanism M i , thus obtaining the global memory state M N , M N ={ M 1 , M 2 ,... M i}, where i=1,2,...,N;

[0157] Recursively update memory and initialize memory state during the memory construction phase It is an all-zero matrix and updates the memory segment by segment. The number of memory states changes according to the sequence of video segments. The number of video segments is N, that is, the first video segment The corresponding memory status is M 1 , video segment 2 The corresponding memory status is M 2 ; Recursively update the memory state segment by segment, the i-th video segment The corresponding memory status is M i ; Finally get the global memory state M N , M N ={ M 1 , M 2 ,... M i}, i=1,2,...,N;

[0158] For the i-th video segment, flatten its corresponding enhanced latent code And generate the query vector through linear transformation , according to the previous memory state M i-1 Generate new memory states using attention mechanism M new , and then the previous memory state is weighted and merged through the gated fusion coefficient g M i-1 With the new memory state M new , generate the current memory state ;

[0159] The formula for calculating the attention mechanism is as follows: ; Among them, softmax() represents the normalized exponential function;

[0160] The gated fusion coefficient g is calculated by the following formula: ;in, Indicates that the Sigmoid function maps real input to the (0,1) interval; linear() indicates a linear transformation function;

[0161] S44, enhance the potential coding corresponding to the i-th video segment The corresponding memory status M i After flattening and concatenation, the segmented latent vector is output through linear dimensionality reduction , and obtain the segmented latent vector sequence ;

[0162] Finally, the enhanced latent code of each video segment is Corresponding memory status Flatten and concatenate to obtain the context-aware latent vector, and then use linear dimensionality reduction to output the segmented latent vector. , and the global memory state .

[0163] Output: sequence of segmented latent vectors and the final global memory state The segmented latent vector will be used as the input for subsequent character-scene decoupling generation, while the memory state Preserve cross-segment long-term dependency information and provide dynamic context support for subsequent autoregressive generation.

[0164] S5, character-scene decoupling generation: the segmented latent vector sequence is generated by noise injection, and the noisy latent vector is generated by the improved 3D-UNet for decoupled perception. The DDIM deterministic sampling strategy is then used to iteratively denoise the noisy latent vector to obtain the final sampled output. Finally, the final sampled output is input into the symmetric 3D convolution decoder, and the RGB video segment sequence is reconstructed by deconvolution upsampling;

[0165] The input of step S5 includes: segmented latent vector sequence , global memory status , compact character identity vector , scene semantic compact vector and text features .

[0166] Step S5 specifically includes the following steps:

[0167] S51, noise injection and conditional initialization, first of all, the segmented potential vector sequence Noise injection: superimpose time-varying Gaussian noise with linearly increasing time steps t , generating a noise latent vector , ;in is the pollution noise scheduling coefficient of time step t, which is used to pollute data (noise); time step t is the specific number of steps in a single noise addition operation, which is set to 50 by default and can be changed according to actual needs; and based on the role identity compact vector , scene semantic compact vector and text features Synchronously initialize the three path conditions to generate identity key-value pairs , scene key-value pairs and conditional embedding features ;

[0168] Initialize the three path conditions simultaneously, including:

[0169] Role Regulation Path: Role Identity Compact Vector Warp Identity Projection Matrix Mapped to identity key-value pairs ;

[0170] Scene Control Path: Scene Semantic Compact Vector Scene projection matrix Mapped to scene key-value pairs ;

[0171] Text-guided paths: Text features Converted into conditional embedding features through cross-attention layer , ;

[0172] Identity Projection Matrix with the scene projection matrix By random initialization assignment;

[0173] In machine learning, projection matrices are often used for dimensionality reduction to reduce data complexity and improve computational efficiency. When a vector is transformed using a projection matrix, the result is a new vector that represents certain features of the original vector in a lower-dimensional space. This new vector can be viewed as a series of key-value pairs, where each key corresponds to a dimension in the projected space, and the value is the coordinate or feature value along that dimension. This is prior art and will not be further elaborated here.

[0174] S52, Decoupled Perceptual UNet Forward Propagation, Noise Latent Vector Input improved 3D-UNet according to identity key-value pairs , scene key-value pairs and conditional embedding features Perform decoupling perception to generate character path features and scene path characteristics ;

[0175] The improved 3D-UNet is divided into an encoding module and a decoding module. The encoding module has 13 layers of convolutional feature extraction blocks;

[0176] In the character control path, the intermediate features corresponding to the 3rd, 7th, and 11th layers of 3D-UNet (character control prioritizes shallow layers, where the 3rd layer (64×64) captures fine movements and the 7th and 11th layers (32×32) model motion trajectories) Generate query vector , generate character attention through attention calculation , , ; Character Attention Update features through residual connections to inject character details: , It will be used as the input of the next layer and finally propagated to the end through multiple layers of convolution to generate the character path feature. : ,ConvBlock represents the convolutional feature extraction block;

[0177] In the scene control path, the intermediate features corresponding to the 5th, 9th, and 13th layers of 3D-UNet (scene control focuses on the middle layer: the 5th layer (32×32) captures local interactions, and the 9th and 13th layers (16×16) model the global context) Generate query vector , generate scene attention through attention calculation , , , scene attention Update features through residual connections to inject scene details: , It will be used as the input of the next layer and finally propagated to the end through multiple layers of convolution to generate scene path features. : ,ConvBlock represents the convolutional feature extraction block, and the parameters and role branches in the scene control path are strictly isolated to ensure decoupling reliability;

[0178] in is the intensity coefficient adjusted for the verb phrase; is the intensity coefficient adjusted by the scene phrase; To query the projection matrix, randomly initialize the values;

[0179] Conditional embedding features are processed via standard crisscross attention layers Perform global guidance, where action phrases are dynamically regulated Value, dynamic control of environmental description phrases value.

[0180] The standard cross-attention layer is part of the improved 3D-UNet. The action phrase is a subset of the text features that describes the character's dynamic behavior (such as "running" and "waving"), and the character path features are dynamically controlled through the value path of the cross-attention layer. The environmental description phrase is a subset of scene-related descriptions extracted from text features (such as "snow mountain" and "cafe"). The scene path features are regulated by the value path of another set of cross-attention layers. .

[0181] S53, memory-enhanced feature fusion, through global memory Enhancing cross-segment consistency: Global memory state is extended to the spatial dimension after projection , and the skip connection feature of 3D-UNet Splicing as memory splicing feature , and finally through the convolution kernel Implement fusion to generate memory fusion features ,in is the convolution function; The weight of ;

[0182] Skip connection is one of the characteristics of UNet itself. 3D-UNet is an improved network of UNet and also has the skip connection feature. This is an existing technology and will not be described here.

[0183] It's understandable that matrices (multidimensional tensors) are the fundamental form of weights in neural networks. Whether it's a fully connected layer, a convolutional layer, or an attention mechanism, weights exist in the form of matrices (or multidimensional tensors) and participate in mathematical operations (such as matrix multiplication and convolution).

[0184] Memory projection matrix and All are randomly initialized and assigned values;

[0185] S54, 3D-UNet terminal feature synthesis, the decoder terminal receives three feature inputs: character path feature , scene path features and memory fusion features Generating prediction noise of 3D-UNet through feature synthesis layer : ,in is the GeLU activation function, is the hidden layer projection matrix, is the output layer weight;

[0186] The feature synthesis layer (feature fusion through skip connections) is the core component of 3D-Unet. It realizes multi-scale feature fusion through skip connections between the encoder and decoder, which is crucial for preserving spatial information and improving segmentation accuracy.

[0187] S55, deterministic sampling, based on predicted noise DDIM sampling is used to sample the noise latent vector Perform iterative denoising to obtain the final sample output : ,in is the denoising scheduling coefficient at time step t, which is used to purify data. It is the timing control valve of DDIM sampling, used to control the noise mixing ratio at the current time step t and balance the weight of data reconstruction and noise retention; is the denoising scheduling coefficient after one iteration of time step t; is the noise latent vector The tentative vector after one step of repair is the intermediate value;

[0188] if is a severely damaged old photo (the larger t is, the more serious the damage is), and To repair the tentative effect after one step, continue to reduce t for iterative denoising repair until t becomes 0, and then the final sampling output is obtained. .

[0189] DDIM (Denoising Diffusion Implicit Models) is an efficient sampling algorithm for diffusion models. Its core is to achieve denoising through deterministic mapping, significantly improving generation speed and sample quality. Convert to clean latent representation The task will finally As the final sample output.

[0190] S56, video reconstruction, the final sampling output Input symmetric 3D convolution decoder, reconstructed into RGB video segment through deconvolution upsampling , get the RGB video segment sequence .

[0191] The symmetric 3D convolutional decoder is a neural network architecture that combines 3D convolution with symmetric design principles. It is primarily used for reconstructing or generating spatiotemporal data such as videos and medical images. Its core approach is to capture the spatiotemporal characteristics of the data by sliding the 3D convolution kernel across the three dimensions of time, height, and width. A symmetric encoder-decoder architecture is used to achieve a balance between feature extraction and data reconstruction.

[0192] The final output is the reconstructed RGB video segment sequence , whose dimensions are unified as (number of frames × height × width × channels), in the role identity consistency (by Guarantee), scene dynamic coherence (by driven), action semantic accuracy (by Control) to meet the controllable generation requirements, and achieve seamless segmented splicing through boundary constraints, laying the foundation for the final long video synthesis.

[0193] Step S5 is the core of multimodal diffusion:

[0194] S51 is noise injection diffusion, which adds time-varying noise to the segmented potential vector and starts the diffusion process;

[0195] S52 is multimodal conditional denoising. The improved 3D-UNet (the core of the diffusion model) receives noisy input and uses the action phrases (controlling the role) and environment description phrases (controlling the scene) decoupled from the text to guide denoising through a cross-attention layer;

[0196] S55 is DDIM sampling, which uses diffusion model efficient sampling technology to generate the final output.

[0197] This step achieves the following by fusing multimodal conditions with the diffusion model:

[0198] 1) High-definition image quality: Progressive optimization of the diffusion model brings detailed realism far exceeding traditional methods.

[0199] 2) Refined independent control: Action / environment phrases dynamically decouple and regulate roles and scenes through a diffusion process.

[0200] 3) Long time series are more stable: Diffusion generation combined with memory mechanism significantly reduces breakage / flickering in long videos.

[0201] S6, block autoregressive generation, segmenting RGB video into sequences and global memory status The final long video is synthesized after segment alignment and temporal interpolation enhancement. ;

[0202] RGB video segment sequence and global memory status As the core input, the final long video is generated through segment alignment and temporal interpolation enhancement. ; Each segment , whose dimensions are unified as (number of frames × height × width × channels).

[0203] The segment alignment specifically includes:

[0204] It is understandable that each video segment contains 16 frames of video images. If the index is 1 to 16, the video segment The first two frames Represents the first two frames of a 16-frame video image, namely frame 1 and frame 2; video segmentation The last two frames Represents the last two frames of the 16-frame video image, namely the 15th and 16th frames;

[0205] Similarly, video segmentation The first two frames Represents the first two frames of a 16-frame video image, namely frame 1 and frame 2; video segmentation The last two frames Represents the last two frames of the 16-frame video image, namely the 15th and 16th frames;

[0206] S61, first extract motion features, for adjacent video segments and , first extract video segments through pre-trained I3D network The last two frames Movement characteristics and extract video segments The first two frames Movement characteristics , where I3D() represents the pre-trained I3D network;

[0207] Then calculate the alignment loss , , the first term of this alignment loss function measures the difference between the motion features of adjacent segments, and the second term measures the difference in temporal gradients of adjacent frames, with the aim of making adjacent segments more coherent in content and motion, where, Represents L2 norm operation; Represents L1 norm operation; Represents the temporal gradient operator, which is used to calculate the change in the video frame features in the temporal dimension (i.e., the feature difference between adjacent frames). Its value is automatically obtained by directly calculating the difference between the features of adjacent frames; ;

[0208] Then according to the alignment loss Boundary optimization: By fixing Update at the same time , Implement boundary optimization and update the optimized boundary frame to the corresponding video segment and , then segment the video and As a splicing feature By aligning the linear matrix Mapping to alignment space to get alignment segments , and finally output the aligned segment sequence , whose boundary frame satisfies Smooth transition conditions; It is the learning rate of the boundary position correction step, which is used to control the optimization amplitude of the segment boundary position. The default value is 0.001;

[0209] Align linear matrices Random initialization assignment;

[0210] Align Segments yes and The result of fusion, aligned linear matrix Join features Mapped to alignment space.

[0211] It is the supervision goal of aligning features across segments, providing the model with a standard answer for cross-segment action coherence, and forcing the model to learn a reasonable transition pattern between adjacent segments. For example: if the i-th segment is "raising the hand" and the i+1-th segment is "swinging a fist", then The feature of "transition from hand raising to fist throwing" should be coded.

[0212] The last two frames and the first two frames are key frames on the temporal boundaries of adjacent video segments. By processing and optimizing them, we can ensure that the generated video is natural and smooth at the segment connections.

[0213] The I3D network is a video action recognition model proposed by Google DeepMind in 2017 (and a classic model). Its core idea is to "expand" 2D image convolution kernels into 3D to directly learn spatiotemporal features. Its architecture is a two-stream: an RGB stream processes raw frames, capturing both appearance and motion; and an optical flow stream, which takes in optical flow maps and explicitly models inter-frame motion. Its function is to extract spatiotemporal features from video clips (encoding both appearance and motion information).

[0214] Here, the pre-trained I3D model is directly called (not retrained) to extract the motion features corresponding to the first and last two frames (such as the output of the optical flow branch). The output features are used for subsequent behavior recognition or time series modeling.

[0215] S62, in the alignment segment The motion-aware neural network interpolation algorithm is used to evenly insert 4 intermediate frames and output an enhanced segmented sequence. ;

[0216] This embodiment uses motion-aware neural network interpolation to align the segments Evenly insert 4 intermediate frames (the interpolation positions are 0.2T, 0.4T, 0.6T, and 0.8T, which are dynamically calculated according to the total number of frames T and rounded up to the nearest integer), ensuring that all interpolation points are valid integer frame positions). After each video segment is inserted from the original 16 frames into M frames, the segment is expanded to 20 frames, and a 20-frame enhanced segment sequence is output. , achieving a frame rate increase to 60fps;

[0217] Motion perception neural networks model and process motion information in videos by simulating the neural network structure associated with motion perception in the human brain. In video interpolation tasks, these networks generate intermediate frame features through optical flow estimation, feature warping, and CNN fusion, achieving high-quality interpolation between frames.

[0218] In this embodiment, the motion perception neural network interpolation algorithm adopts DAIN;

[0219] Depth-Aware Video Frame Interpolation (DAIN), developed by Wenbo Bao et al. and first presented at CVPR 2019, builds on depth perception technology by detecting depth cues to explicitly identify occluded areas and develops a depth-aware flow projection layer to synthesize intermediate flows that prioritize sampling closer objects.

[0220] DAIN introduces a depth map to distinguish foreground from background, solving occlusion issues and generating more reasonable intermediate frames. Based on optical flow estimation, the DAIN model further considers the accelerated motion of objects, improving the accuracy and stability of interpolated frames.

[0221] In another possible implementation, the segment sequence is aligned As input, first segment based on alignment The optical flow amplitude of each frame Detect the key frames corresponding to the intense motion intervals, and then use the interpolator Interpolate the feature vector of the key frame corresponding to the intense motion interval with the feature vector of the next frame to generate the feature vector of the intermediate frame ,in, Align segments The feature vector of the kth frame in Align segments The feature vector of the k+1th frame is then converted into an actual pixel frame through the output bidirectional optical flow field of the optical flow estimation network. , the actual pixel frame Insert alignment segment directly Get enhanced segment sequence , filling in the action details between key frames to enhance the smoothness of movement.

[0222] The interpolator U-Net is used to extract spatiotemporal features and global memory Linear projection Fusion to skip connection features;

[0223] Optical flow is the instantaneous speed of pixel movement of a moving object in space on the observation imaging surface. It uses the change of pixels in the time domain of the image sequence and the correlation between adjacent frames to find the corresponding relationship between the previous frame and the current frame, thereby calculating the motion information of the object between adjacent frames. It reflects the intensity of pixel motion. When an object moves quickly in the video, the optical flow amplitude of the corresponding pixel will be larger; on the contrary, when the object is stationary or moves slowly, the optical flow amplitude is smaller. Therefore, by setting a violent motion amplitude threshold τ, when When , it can be determined that the area is a violent motion area, and then the starting frame corresponding to the violent motion area is used as the violent motion interval. The violent motion amplitude threshold τ can be selected according to the actual situation;

[0224] U-Net is a commonly used convolutional neural network architecture with a symmetrical encoder-decoder structure. It can effectively extract image features and perform upsampling operations, and is suitable for image generation and interpolation tasks.

[0225] The optical flow estimation network is a model based on deep learning, which aims to estimate pixel-level motion information from consecutive frames of a video. It automatically extracts image features and establishes mapping relationships between adjacent frames by learning a large amount of video data, thereby predicting the displacement of each pixel in time. The optical flow estimation network usually takes two or more consecutive frames of images as input, and after processing by multiple convolutional layers, pooling layers and other neural network layers, outputs an optical flow field of the same size as the input image. Each element in this optical flow field represents the motion vector of the corresponding pixel in the horizontal and vertical directions; the bidirectional optical flow field considers the motion information of objects in the video in two opposite directions. Traditional optical flow fields usually only estimate unidirectional motion from the previous frame to the current frame, while the bidirectional optical flow field simultaneously estimates the motion from the previous frame to the current frame and from the current frame to the previous frame, thereby more comprehensively describing the motion of pixels; the optical flow estimation network of this embodiment sets two independent output branches in the network structure, which are used to estimate and output forward and reverse optical flows respectively. After training, it can output bidirectional optical flow field information;

[0226] yes Aligned and optimized frame sequence after S61 motion alignment.

[0227] Synthesize the final long video Specifically include:

[0228] S63, enhanced segmented sequence after integration and interpolation , constructing a complete video stream through time sequence stitching , where the overlapping area of ​​adjacent enhanced segments uses the Hamming window function Perform weighted average fusion Overwrite the complete video stream The corresponding position of the complete video stream Implement global chroma correction to get the final long video with consistent output color ;

[0229] Among them, global chromaticity correction is implemented based on the following formula: , is the chroma correction function of the zth frame, input the frame to be corrected , output the corrected frame , align the reference frame hue; z represents the frame number in the video, 1, 2, ...T.

[0230] The reference frame is usually the first frame (usually the first frame of the segment) by default, which is used as the color reference. In this embodiment, the reference frame is the center frame of the first segment. ; It is the inverse color space transformation function of the reference frame, used to map the hue / saturation characteristics of the reference frame to a standard color space (such as sRGB).

[0231] Global memory In the interpolator It uses the CNN as a conditional input and ensures cross-segment motion continuity through feature fusion, forming a complete optimization link from segment alignment, frame rate improvement to global splicing.

[0232] The last two frames (tail frames) of each adjacent enhanced segment and the first two frames (head frames) of the next segment form an overlapping region (two frames). These four frames in this overlapping region are weighted fused using a Hamming window function, and the result is overwritten into the corresponding position in the final video stream. This operation eliminates segment seams, ensuring motion continuity while preserving detail.

[0233] Steps S62 and S63 are the process of temporal interpolation enhancement.

[0234] This embodiment proposes a long video character scene decoupling generation method based on multimodal diffusion. Its core is to solve the key problems of rough dynamic control, limited generation length and excessive resource consumption in long video generation through multimodal condition decoupling, causal spatiotemporal modeling and branch synthesis technology, thereby significantly improving the quality and efficiency of generated videos.

[0235] This embodiment designs a diffusion generation architecture that independently optimizes characters and scenes, uses text-driven multimodal conditions to separate and control character behavior and background dynamics; introduces causal 3D convolution and a global memory mechanism to maintain cross-block temporal consistency; this embodiment of the invention can support the demand for efficient and controllable long video generation in scenarios such as film and television pre-visualization and virtual character driving, breaking through the bottleneck of existing methods in decoupling editing capabilities and generation length.

[0236] This embodiment provides content creators who have needs for storyboard editing and virtual production in the film and television industrialization process with a long video generation tool that combines high-precision control with low resource consumption.

[0237] Example 2

[0238] refer to Figure 2 According to this embodiment, a long video character scene decoupling generation system based on multimodal diffusion is disclosed, including the following modules:

[0239] An advanced feature extraction module is used to extract high-level semantic features and fine-grained texture features of the input image through the SigLIP encoder and the DINOv2 encoder, and to extract high-level semantic features and fine-grained texture features of the input video through the SigLIP encoder and the DINOv2 encoder, and to synthesize the high-level semantic features and fine-grained texture features in proportion;

[0240] The feature fusion and compression module is used to perform cross-modal feature fusion on high-level semantic features and fine-grained texture features to obtain joint features, and compress the joint features using the Q-Former projection head through the character branch to obtain a compact character identity vector, and compress the joint features using the Q-Former projection head through the scene branch to obtain a compact scene semantic vector;

[0241] The text feature extraction module is used to encode the text prompt P through the T5-XL model to generate text features;

[0242] The block-wise latent diffusion module is used to segment the input video into time series and generate latent codes through a causal 3D convolutional encoder. The character identity compact vector, scene semantic compact vector, and text features are embedded into the latent space through a linear projection matrix and fused with the latent code to obtain an enhanced latent code. The enhanced latent code is then concatenated with the memory state updated by the attention mechanism and the dimensionality is reduced to obtain a segmented latent vector sequence, which then outputs the global memory state.

[0243] The decoupled reconstruction video module is used to generate a segmented latent vector sequence through noise injection. The noisy latent vector is generated through the improved 3D-UNet for decoupled perception. The DDIM deterministic sampling strategy is then used to iteratively denoise the noisy latent vector to obtain the final sampled output. Finally, the final sampled output is input to a symmetric 3D convolutional decoder, and the RGB video segment sequence is reconstructed through deconvolution upsampling.

[0244] The block autoregressive generation module is used to synthesize the final long video by strengthening the RGB video segment sequence and the global memory state through segment alignment and temporal interpolation.

[0245] The long video character scene decoupling generation system based on multimodal diffusion provided by the embodiment of the present invention can execute the long video character scene decoupling generation method based on multimodal diffusion provided by any embodiment of the present invention, and has the functional modules and beneficial effects corresponding to the execution method. It is worth noting that in the above-mentioned embodiment of the long video character scene decoupling generation system based on multimodal diffusion, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be realized; in addition, the specific names of the various functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0246] Example 3

[0247] refer to Figure 3 , Figure 3Schematic diagram of the structure of the device for decoupling and generating character scenes in long videos based on multimodal diffusion in this embodiment. The device 20 for decoupling and generating character scenes in long videos based on multimodal diffusion in this embodiment includes a processor 21, a memory 22, and a computer program stored in the memory 22 and executable on the processor 21. When the processor 21 executes the computer program, it implements the steps in the aforementioned method embodiments. Alternatively, when the processor 21 executes the computer program, it implements the functions of the modules / units in the aforementioned apparatus embodiments.

[0248] Exemplarily, the computer program can be divided into one or more modules / units, which are stored in the memory 22 and executed by the processor 21 to implement the present invention. The one or more modules / units can be a series of computer program instruction segments capable of performing specific functions, which are used to describe the execution process of the computer program in the multimodal diffusion-based long video character scene decoupling generation device 20. For example, the computer program can be divided into the modules described in Example 2. For the specific functions of each module, please refer to the operating process of the device described in the above embodiment, and will not be repeated here.

[0249] The long video character scene decoupling generation device 20 based on multimodal diffusion may include, but is not limited to, a processor 21 and a memory 22. Those skilled in the art will understand that the schematic diagram is merely an example of the long video character scene decoupling generation device 20 based on multimodal diffusion, and does not constitute a limitation of the long video character scene decoupling generation device 20 based on multimodal diffusion. The device may include more or fewer components than shown in the figure, or combine certain components, or different components. For example, the long video character scene decoupling generation device 20 based on multimodal diffusion may also include input and output devices, network access devices, buses, etc.

[0250] The processor 21 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. The processor 21 is the control center of the multimodal diffusion-based long video character scene decoupling generation device 20, and utilizes various interfaces and lines to connect the various parts of the multimodal diffusion-based long video character scene decoupling generation device 20.

[0251] The memory 22 can be used to store the computer programs and / or modules. The processor 21 implements the various functions of the multimodal diffusion-based long video character scene decoupling generation device 20 by running or executing the computer programs and / or modules stored in the memory 22 and accessing the data stored in the memory 22. The memory 22 may primarily include a program storage area and a data storage area. The program storage area may store an operating system and at least one application required for a function (such as sound playback or image playback); the data storage area may store data generated based on the use of the mobile phone (such as audio data and a phone book). Furthermore, the memory 22 may include high-speed random access memory (RAM) and non-volatile memory, such as a hard disk, internal memory, a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other volatile solid-state storage device.

[0252] If the modules / units integrated in the multimodal diffusion-based long video character scene decoupling generation device 20 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the present invention can also implement all or part of the processes in the above-mentioned method embodiments by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When executed by the processor 21, the computer program can implement the steps of each of the above-mentioned method embodiments. The computer program includes computer program code, which can be in source code form, object code form, executable file, or some intermediate form. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, software distribution medium, etc. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practices in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practices, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0253] It should be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. In addition, in the drawings of the device embodiments provided by the present invention, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines. A person of ordinary skill in the art can understand and implement the present invention without inventive effort.

[0254] The parts of the present invention that are not described in detail are prior art. It is obvious to those skilled in the art that the present invention is not limited to the details of the above-mentioned exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or basic characteristics of the present invention. Therefore, no matter from which point of view, the embodiments should be regarded as exemplary and non-restrictive, and it is intended that all changes that fall within the meaning and scope of equivalent elements are included in the present invention.

Claims

1. A method for decoupling and generating character scenes in long videos based on multimodal diffusion, characterized by: The following steps are involved: S1. Extract high-level semantic features and fine-grained texture features of the input image through the SigLIP encoder and the DINOv2 encoder, and extract high-level semantic features and fine-grained texture features of the input video through the SigLIP encoder and the DINOv2 encoder, and synthesize the high-level semantic features and fine-grained texture features in proportion. S2. Perform cross-modal feature fusion on high-level semantic features and fine-grained texture features to obtain joint features, and compress the joint features using the Q-Former projection head through the character branch to obtain a compact character identity vector, and compress the joint features using the Q-Former projection head through the scene branch to obtain a compact scene semantic vector; S3, text prompt P generates text features through T5-XL model encoding; S4: After temporal segmentation of the input video, a latent code is generated through a causal 3D convolutional encoder. The character identity compact vector, scene semantic compact vector, and text features are embedded into the latent space through a linear projection matrix and fused with the latent code to obtain an enhanced latent code. The enhanced latent code is then concatenated with the memory state updated by the attention mechanism and the dimensionality is reduced to obtain a segmented latent vector sequence, and the global memory state is output. S5. The noise latent vector sequence is injected into the segmented latent vector, and the noise latent vector is generated by the improved 3D-UNet for decoupled perception. The DDIM deterministic sampling strategy is then used to iteratively denoise the noise latent vector to obtain the final sampled output. Finally, the final sampled output is input into the symmetric 3D convolution decoder, and the RGB video segment sequence is reconstructed by deconvolution upsampling. S6. The RGB video segment sequence and the global memory state are enhanced by segment alignment and temporal interpolation to synthesize the final long video.

2. The method according to claim 1, characterized in that Step S1 specifically includes the following steps: S11, input image , and perform scaling normalization to obtain the normalized image ; Where H is the image height, W is the image width, the unit is pixel; 3 is the RGB three channels; S12, input video Get a set of key frames by uniform sampling V K ={ V 1 , V 2 ,..., V k }, and for each keyframe V k Perform scaling normalization to obtain , thus obtaining a normalized key frame set ; T is the number of video frames; S13. Extract normalized image through SigLIP encoder High-level semantic features of images , , and extract the normalized image through the DINOv2 encoder Fine-grained texture features of images , ; S14. Extracting normalized key frame sets through SigLIP encoder The high-level semantic features of each frame are pooled and the high-level semantic features of the video are obtained by temporal average pooling. , and extract the normalized keyframe set through the DINOv2 encoder The fine-grained texture features of each frame of the image are pooled and the video fine-grained texture features are obtained by temporal average pooling. ; S15, high-level semantic features of the image High-level semantic features of videos Proportionally synthesize high-level semantic features , and the fine-grained texture features of the image Fine-grained texture features of videos Proportional synthesis of fine-grained texture features .

3. The method according to claim 1, characterized in that Step S2 specifically includes the following steps: S21, high-level semantic features Broadcasting to spatial dimensions and fine-grained texture features Splicing along the channel dimension and performing cross-modal optimization through two layers of Transformer to obtain joint features , ; S22, the learnable query vector of the role branch is combined with the joint feature through the cross attention layer After interaction, the self-attention Transformer block integrates global information and linear projection, and then uses the Q-Former projection head to compress the character identity compact vector ; S23, the learnable query vector of the scene branch is combined with the joint feature through the cross attention layer After interaction, the self-attention Transformer block integrates global information and linear projection, and then uses the Q-Former projection head to compress the scene semantics compact vector .

4. The method according to claim 1, wherein Step S4 specifically includes the following steps: S41, input video Cut into N video segments of fixed length L , each video segment The latent code is generated by processing it through a causal 3D convolutional encoder z i ; S42, role identity compact vector , scene semantic compact vector and text features Embedded into the latent space through the linear projection matrix and compared with the latent code z i Fusion Enhanced Latent Code ; S43, the memory state corresponding to the i-1th video segment M i-1 and enhanced latent coding The memory state corresponding to the updated i-th video segment is calculated segment by segment through the attention mechanism M i , thus obtaining the global memory state M N , M N ={ M 1 , M 2 ,..., M i }, where i=1,2,...,N; S44, enhance the potential coding corresponding to the i-th video segment The corresponding memory status M i After flattening and concatenation, the segmented latent vector is output through linear dimensionality reduction , and obtain the segmented latent vector sequence .

5. The method according to claim 1, wherein Step S5 specifically The following steps are involved: S51, first segment the potential vector sequence Noise injection: superimpose time-varying Gaussian noise with linearly increasing time steps t , generating a noise latent vector , ;in is the pollution noise scheduling coefficient at time step t; and according to the role identity compact vector , scene semantic compact vector and text features Synchronously initialize the three path conditions to generate identity key-value pairs , scene key-value pairs and conditional embedding features ; S52, Noise Latent Vector Input improved 3D-UNet according to identity key-value pairs , scene key-value pairs and conditional embedding features Perform decoupling perception to generate character path features and scene path characteristics ; S53, the global memory state is extended to the spatial dimension after projection , and the skip connection feature of 3D-UNet Splicing as memory splicing feature , and finally through the convolution kernel Implement fusion to generate memory fusion features ,in is the convolution function; The weight of ; S54, Role Path Characteristics , scene path features and memory fusion features Generating prediction noise of 3D-UNet through feature synthesis layer : ,in is the GeLU activation function, is the hidden layer projection matrix, is the output layer weight; S55, based on the predicted noise DDIM sampling is used to sample the noise latent vector Perform iterative denoising to obtain the final sample output : ,in is the denoising scheduling coefficient at time step t; is the denoising scheduling coefficient after one iteration of time step t; is the noise latent vector The tentative vector after one step of repair; S56, output the final sample Input symmetric 3D convolution decoder, reconstructed into RGB video segment through deconvolution upsampling , get the RGB video segment sequence .

6. The method according to claim 1, characterized in that Step S6 specifically includes the following steps: S61, first extract motion features, for adjacent video segments and , first extract video segments through pre-trained I3D network The last two frames Movement characteristics and extract video segments The first two frames Movement characteristics ; Then calculate the alignment loss , , and then according to the alignment loss Perform boundary optimization to obtain the final output aligned segment sequence ;in, Represents L2 norm operation; Represents L1 norm operation; represents the time gradient operator; ;I3D() represents the pre-trained I3D network; S62, in the alignment segment The motion-aware neural network interpolation algorithm is used to evenly insert 4 intermediate frames and output an enhanced segmented sequence. ; S63, strengthen the segment sequence Constructing a complete video stream through time-series stitching , and the complete video stream Perform global chroma correction and output the final long video with consistent color .

7. The method according to claim 2, characterized in that The scaling normalization process in step S11 is implemented based on the following formula: ； Among them, resize() represents a scaling function.

8. The method according to claim 3, characterized in that Step S22 is specifically as follows: The character branch initializes 16 learnable query vectors , through the cross attention layer and the joint feature interaction, and then integrate the global information through 4 layers of self-attention Transformer blocks, and linearly project it into dimensional role identity vector , and finally use the Q-Former projection head to transform the character identity vector Compressed into a compact character identity vector .

9. The method according to claim 4, characterized in that Step S41 is specifically as follows: The causal 3D convolutional encoder consists of 3 residual blocks, 3 convolutional layers, and 2 pooling layers; the input is a video segment , the output is the potential code corresponding to the video segment z i ; Input video segment First, the causal 3D convolutional encoder is used for layer-by-layer processing: the first convolutional layer uses a kernel size of The convolution of the network outputs a 64-channel feature map. The time axis is padded with 1 frame on the left to ensure causality, and the space is padded with 3 pixels to maintain resolution. The activation function is GELU to obtain the intermediate feature. ; The second layer passes through the kernel size , convolution with 128 channels, time step 2, downsampling with spatial step 2 to a resolution of 128×128, and applying a temporal causal mask, with an output dimension of ; The third layer uses a kernel size of , convolution with 256 channels, further downsampled to 64×64, to obtain the feature map ; Subsequently, the features are optimized through three residual blocks ResBlock: each residual block contains GroupNorm, GELU activation, causal 3D convolution and skip connection to keep the feature dimension unchanged; average pooling is applied to the processed features along the time axis to compress the time step from 8 to 4, and the feature map is obtained. Finally, the number of channels is maintained through 1×1×1 convolution, and the resolution is reduced to 32×32 in combination with spatial maximum pooling to generate the final low-resolution potential code .

10. A long video character scene decoupling generation system based on multimodal diffusion, characterized by: Includes the following modules: An advanced feature extraction module is used to extract high-level semantic features and fine-grained texture features of the input image through the SigLIP encoder and the DINOv2 encoder, and to extract high-level semantic features and fine-grained texture features of the input video through the SigLIP encoder and the DINOv2 encoder, and to synthesize the high-level semantic features and fine-grained texture features in proportion; The feature fusion and compression module is used to perform cross-modal feature fusion on high-level semantic features and fine-grained texture features to obtain joint features, and compress the joint features using the Q-Former projection head through the character branch to obtain a compact character identity vector, and compress the joint features using the Q-Former projection head through the scene branch to obtain a compact scene semantic vector; The text feature extraction module is used to encode the text prompt P through the T5-XL model to generate text features; The block-wise latent diffusion module is used to segment the input video into time series and generate latent codes through a causal 3D convolutional encoder. The character identity compact vector, scene semantic compact vector, and text features are embedded into the latent space through a linear projection matrix and fused with the latent code to obtain an enhanced latent code. The enhanced latent code is then concatenated with the memory state updated by the attention mechanism and the dimensionality is reduced to obtain a segmented latent vector sequence, which then outputs the global memory state. The decoupled reconstruction video module is used to inject noise into the segmented latent vector sequence to generate a noisy latent vector. The noise latent vector is then decoupled and generated through an improved 3D-UNet. The DDIM deterministic sampling strategy is then used to iteratively denoise the noise latent vector to obtain the final sampled output. The final sampled output is input into a symmetric 3D convolutional decoder, and the RGB video segment sequence is reconstructed through deconvolution upsampling. The block autoregressive generation module is used to synthesize the final long video by strengthening the RGB video segment sequence and the global memory state through segment alignment and temporal interpolation.

Images